robots.txt
robots.txt is a small text file at the root of a site that tells automated crawlers, politely and by request rather than by force, which parts of the site they should and should not visit.
robots.txt is one of the oldest conventions on the web, a plain text file sitting at yoursite.com/robots.txt that lists instructions for automated crawlers: which sections of the site they are welcome to visit, and which they should leave alone. Search engines, AI crawlers, and countless other bots check it before they start.
It works entirely on trust. There is no lock on the folders it mentions, no technical enforcement at all. It is a request, honoured by well-behaved crawlers and ignored by the ones that were never going to respect a boundary anyway.
Why it matters to you
Most small business sites do not need much in this file, and the biggest risk is having it configured wrong rather than not having it at all. A misconfigured robots.txt that accidentally blocks your entire site from search engines is a real, if rare, way to disappear from search overnight, usually inherited from a template or a developer testing something and forgetting to undo it.
Used correctly, it steers crawlers away from parts of the site that add no value to search, admin areas, internal search result pages, duplicate content generated by filters, so crawlers spend their limited attention on the pages that actually matter.
How I set it up
I keep it simple and deliberate. On the sites I build, it points crawlers to the sitemap, allows everything that should be found, and blocks only the handful of paths that genuinely add nothing to search, like admin routes or internal tooling. I do not use it to try to hide content from the public, because it cannot actually do that.
I also check it whenever something unexpected happens with indexing, because a wrongly blocked section is one of the first things worth ruling out. It is a small file, and small files are exactly the kind of thing that gets edited once during a migration and never looked at again.
What it looks like in practice
You will likely never open this file, and that is fine. If you do, it is a handful of plain lines, a couple of “allow” and “disallow” rules and a pointer to the sitemap. Nothing dramatic, nothing that should change often.
The only time it becomes visible to you at all is if something is missing from search unexpectedly and this file turns out to be the reason, which is uncommon on a well-built site but worth ruling out first because it is quick to check.
Questions I get about this
- Does robots.txt keep a page out of Google's search results?
- Not reliably on its own. It asks crawlers not to visit a page, but if other sites link to that page, Google can still list it in search results without having crawled the content. Keeping something out of search results properly needs a different tool, not robots.txt.
- Can I use it to hide private information?
- No, and this trips people up. The file itself is public. Anyone can open yoursite.com/robots.txt and read exactly which folders you asked crawlers to skip, which is sometimes a map of exactly what you did not want found. Never rely on it for anything actually sensitive.
- Do well-behaved crawlers always obey it?
- Major search engines do, as a rule. It is a voluntary standard, not an enforced barrier, so a crawler that ignores it faces no technical block, only a reputational one. Most of the crawlers worth caring about behave.
Want this set up properly for your business?
This is the kind of thing I build every week. Grab a time and we will talk through what fits.