RobotsTxt Pro

Robots.txt best practices

Twelve rules that prevent nearly every robots.txt disaster we see in the validator, from accidental site-wide blocks to CSS that Google can’t load.

  1. Serve it at the root, as plain text. https://example.com/robots.txt, UTF-8, Content-Type: text/plain. Each subdomain and protocol needs its own file.
  2. Return 200 or 404, never 5xx. A 404 means “no restrictions”. A 5xx or timeout makes Google pause crawling the whole site.
  3. Keep it under 500 KiB. Google ignores everything past that. If you’re close, you’re listing URLs instead of patterns.
  4. Don’t block CSS, JavaScript or images your pages need. Google renders pages like a browser and can misjudge pages it can’t render.
  5. Don’t use robots.txt to hide pages from search. Use noindex. Blocked URLs can still be indexed from links.
  6. Don’t use it for security. The file is public and lists exactly what you’d rather people didn’t look at. Use authentication.
  7. Remember specific groups replace *. A User-agent: Googlebot group means Googlebot ignores every * rule. Repeat shared rules.
  8. Use patterns for crawl traps. Faceted filters, sort orders, session IDs and internal search can generate millions of URLs: Disallow: /*?sort=.
  9. Add your sitemap with a full URL.
  10. Decide on AI crawlers deliberately. Training and AI search are separate tokens. See the AI crawler list.
  11. Comment your intent. # Block faceted navigation (added 2026-03) saves the next person from guessing.
  12. Test before and after every change. Run important URLs through the tester and fetch the live file in the validator after deploys.

The launch-day check

The most expensive robots.txt bug is a staging Disallow: / that ships to production. After every launch or migration, fetch /robots.txt on the live domain and confirm the * group doesn’t block /.

Most robots.txt disasters start with a rule nobody pictured from the crawler’s side. Picture things from somewhere else and plenty of puzzles clear up, like why the Moon has phases, which ahaboo narrates step by step.

Questions

Should every site have a robots.txt?

It’s optional; without one, everything may be crawled. A short file with a Sitemap line is still worth having, and it stops your logs filling with 404s for /robots.txt.

How often do crawlers re-read robots.txt?

Google generally caches it for up to 24 hours. Other crawlers vary; OpenAI and Meta also mention about 24 hours.

Sources: Google: Create and submit a robots.txt file; Google: robots.txt specification; RFC 9309.