Allow all crawlers everywhere
Use when: You want everything crawled. An empty Disallow (or no robots.txt at all) means the same thing.
An empty Disallow value is ignored, so nothing is disallowed.
User-agent: *
Disallow:
Robots.txt examples
Ten patterns that cover most real sites. Copy one as-is or open it in the generator to combine it with others.
Block all crawlers from the whole site
Use when: Staging sites, private previews, or a site that isn’t ready.
"/" matches every URL. Pages can still appear in search as bare URLs if other sites link to them; use noindex or authentication to keep them out entirely.
User-agent: *
Disallow: /
Block one folder
Use when: Admin areas, internal tools, duplicate print versions.
Matches /admin/ and everything below it, but not /admin (no slash) or /administrator.
User-agent: *
Disallow: /admin/
Block one crawler, allow everyone else
Use when: A single bot is causing trouble or you don’t want it.
CCBot matches its own group and obeys only that; every other crawler falls through to "*".
User-agent: CCBot
Disallow: /
User-agent: *
Disallow:
Opt out of AI training, stay in AI search
Use when: You want to appear in ChatGPT, Claude and Perplexity search results but keep content out of model training.
Stacked user-agent lines share one group. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot aren’t named, so they follow "*".
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: *
Disallow:
Block URLs with query parameters
Use when: Faceted navigation, session IDs or sort orders creating endless URL variants.
"*" matches any characters, so /*?sort= catches ?sort= on any path. Add the "&" form for parameters that can appear later in the query string.
User-agent: *
Disallow: /*?sort=
Disallow: /*?sessionid=
Disallow: /*&sessionid=
Block a file type
Use when: Keep crawlers away from PDFs or other downloads.
"$" anchors the end of the URL, so /report.pdf is blocked but /report.pdf?download=1 is not.
User-agent: *
Disallow: /*.pdf$
Allow one page inside a blocked folder
Use when: A public file lives inside an otherwise private folder.
The Allow rule is longer, so it wins for that one URL. Order in the file doesn’t matter to Google.
User-agent: *
Disallow: /private/
Allow: /private/press-kit.pdf
Block internal search results
Use when: Search pages are thin, infinite and waste crawl budget.
Covers /search, /search/anything and any URL whose query starts with q=.
User-agent: *
Disallow: /search
Disallow: /*?q=
Sitemaps only, no restrictions
Use when: You just want crawlers to find your sitemap.
Sitemap lines stand alone and aren’t tied to any group. List as many as you like, with full URLs.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/news-sitemap.xml
Questions about these examples
What is the simplest valid robots.txt?
"User-agent: *" followed by an empty "Disallow:". It allows everything, which is also what happens if there is no robots.txt at all.
What does "User-agent: * Disallow: /" mean?
Every crawler that has no group of its own is asked not to crawl any URL on the site.
Does the order of Allow and Disallow lines matter?
Not to Google or RFC 9309 parsers: the longest matching path wins regardless of order. Some older crawlers use the first match, so putting Allow exceptions first is a harmless habit.