What is robots.txt?
A robots.txt file is a short text file at the root of a website that tells crawlers which URLs they may request. Think of it as the sign at the car-park entrance: it tells honest drivers where to go, but it can't stop anyone who ignores it.
What it looks like
User-agent: *
Disallow: /admin/
Disallow: /search
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlTwo groups and a sitemap. Every crawler is asked to skip /admin/ and /search; GPTBot is asked to skip everything; and all crawlers are told where the sitemap is.
What robots.txt can and can't do
It can
- Ask compliant crawlers to skip URLs or folders
- Give different rules to different crawlers
- Save crawl budget on filters, search and duplicates
- Point every crawler at your sitemap
- Opt out of documented AI training crawlers
It can't
- Remove a page from search results (use
noindex) - Protect private content (use authentication)
- Force bad bots to comply
- Apply to other subdomains or protocols
- Set crawl speed for Google (
Crawl-delayis ignored)
How a crawler picks its rules
A crawler doesn't read the whole file. It looks for the group whose User-agent names it; if none does, it uses the * group. It follows exactly one of those, never a mix. Choose a crawler to see which lines it actually reads. The file is Google's own example.
- 1user-agent: googlebot-news
- 2disallow: /fish
- 3
- 4user-agent: *
- 5disallow: /carrots
- 6
- 7user-agent: googlebot
- 8disallow: /shrimp
Googlebot-News
Follows: user-agent: googlebot-news
Its own group names it exactly, so it follows that group and nothing else.
Rules it obeys: disallow /fish
Example from Google Search Central: order of precedence for user agents.
Which rule wins inside a group
Once a crawler has its group, it checks the URL's path (and query string) against every Allow and Disallow rule. The longest matching rule wins. If an Allow and a Disallow are exactly as long, Allow wins. The order of lines doesn't matter. The tester lets you step through Google's six worked examples.
The rules of the format
- The file must be at
/robots.txton the exact host and protocol it controls. - UTF-8 plain text; lines end in LF, CR or CR/LF;
#starts a comment. - Field names are case-insensitive. Paths are case-sensitive.
- Supported fields:
user-agent,allow,disallow(RFC 9309) andsitemap(supported by Google, Bing and others). *matches any characters;$at the end anchors the end of the URL.- Google reads only the first 500 KiB.
A short history
Martijn Koster proposed the Robots Exclusion Protocol in 1994, and for 28 years it was an informal convention. In 2019 Google published its parser as open source and submitted the protocol to the IETF, and in September 2022 it became RFC 9309. AI companies have since added their own tokens so publishers can separate search crawling from model training.
Keep reading
- Robots.txt disallow allHow "Disallow: /" and "Allow: /" work, when blocking everything is right, why blocked pages can still show up in Google, and safer options for staging sites.
- Adding a sitemap to robots.txtThe exact Sitemap: syntax, why it must be a full URL, multiple sitemaps and sitemap indexes, and whether it matters where the line goes.
- Noindex in robots.txtGoogle stopped supporting noindex in robots.txt in 2019. Learn why Disallow doesn’t remove pages from search, and how to use meta robots and X-Robots-Tag instead.
- Crawl-delay in robots.txtCrawl-delay is not part of RFC 9309 and Google ignores it. See which crawlers document support, how to write it, and how to actually slow Googlebot down.
- Robots.txt wildcardsTest how * and $ match paths with an interactive pattern checker built on Google’s documented examples, plus common wildcard mistakes.
- Robots.txt best practicesPractical robots.txt rules for SEO: what to block, what never to block, how precedence works, AI crawlers, testing and monitoring.
- How to create a robots.txt fileCreate a robots.txt file in five steps: decide what to block, write the rules, save as UTF-8 plain text, upload to your site root and test it.
Robots.txt basics
What is a robots.txt file?
A plain text file at the root of a website (example.com/robots.txt) that tells automated crawlers which URLs they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022.
Is robots.txt legally binding?
No. It’s a published request. Reputable crawlers follow it; malicious bots can ignore it. Use authentication or server rules for anything that must stay private.
Do I need a robots.txt file?
No. Without one, crawlers may fetch everything. Most sites still benefit from a short file that blocks crawl traps and lists the sitemap.
What’s the difference between robots.txt and a robots meta tag?
robots.txt controls whether a URL is crawled. The robots meta tag (or X-Robots-Tag header) controls whether a crawled page is indexed or shown in results. A crawler must be allowed to fetch a page to see its meta tag.
Where can I see a site’s robots.txt?
Add /robots.txt to the domain, e.g. https://www.google.com/robots.txt. Or fetch it in our validator to get a report.