Block AI crawlers with robots.txt
AI companies use separate tokens for training, for their search index and for pages a user asks about. Choose each one on its merits. Every name below comes from the vendor's own documentation.
# AI crawlers opted out (tokens from each vendor’s documentation)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
Disallow: /
# Everyone else
User-agent: *
Disallow:
Training vs search. Blocking training crawlers keeps future content out of model training. Leaving the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot) allowed keeps you citable in AI answers that link back.
User-triggered fetchers are visits a person asked for. OpenAI, Perplexity, Meta and Amazon each say theirs may not follow robots.txt.
Every documented AI crawler token
| Token | Operator | Type | What it does | Source |
|---|---|---|---|---|
| GPTBot | OpenAI | Model training | Crawls content that may be used to train OpenAI’s generative AI foundation models. | OpenAI docs |
| OAI-SearchBot | OpenAI | AI search index | Surfaces websites in search results in ChatGPT’s search features. | OpenAI docs |
| ChatGPT-User | OpenAI | User-requested fetch | Visits pages when a ChatGPT user or a Custom GPT asks for them. It doesn’t crawl automatically. | OpenAI docs |
| ClaudeBot | Anthropic | Model training | Collects web content that could contribute to training Anthropic’s generative AI models. | Anthropic docs |
| Claude-SearchBot | Anthropic | AI search index | Navigates the web to improve the quality of search results for Claude users. | Anthropic docs |
| Claude-User | Anthropic | User-requested fetch | Fetches pages when a Claude user asks a question that needs them. | Anthropic docs |
| Google-Extended | Usage control token | A product token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. | Google docs | |
| Google-CloudVertexBot | User-requested fetch | Crawls sites whose owners asked for it while building Vertex AI Agents. | Google docs | |
| PerplexityBot | Perplexity | AI search index | Surfaces and links websites in Perplexity search results. | Perplexity docs |
| Perplexity-User | Perplexity | User-requested fetch | Visits pages when a Perplexity user asks a question that needs them. | Perplexity docs |
| Applebot-Extended | Apple | Usage control token | A secondary token that lets publishers opt out of their content being used to train Apple’s foundation models (Apple Intelligence and related services). | Apple docs |
| CCBot | Common Crawl | Model training | Builds Common Crawl’s open web archive, which many AI training datasets are built from. | Common Crawl docs |
| Meta-ExternalAgent | Meta | Model training | Crawls the web for uses such as training foundation AI models or indexing content for Meta products. | Meta docs |
| Meta-ExternalFetcher | Meta | User-requested fetch | Fetches individual links at a user’s request, including for agentic AI features. | Meta docs |
| Amazonbot | Amazon | Model training | Improves Amazon products and services; Amazon says the data may be used to train Amazon AI models. | Amazon docs |
| Amzn-SearchBot | Amazon | AI search index | Improves search experiences in Amazon products such as Alexa. Amazon says it does not crawl for AI training. | Amazon docs |
| Amzn-User | Amazon | User-requested fetch | Fetches pages for actions a user starts. Amazon says it does not crawl for AI training. | Amazon docs |
Checked against each vendor's documentation on 27 September 2026. Tokens appear here only if the operator documents them.
AI crawler questions
How do I block AI crawlers with robots.txt?
Add a group naming each token you want to exclude, e.g. "User-agent: GPTBot" then "Disallow: /". Several User-agent lines can share one Disallow. Use the exact tokens vendors document; made-up names do nothing.
Will blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. OpenAI uses OAI-SearchBot for search and GPTBot for training, and treats them independently. Block GPTBot and allow OAI-SearchBot to opt out of training while staying findable.
Does blocking Google-Extended affect my Google rankings?
No. Google says Google-Extended does not impact a site’s inclusion or ranking in Google Search. It only controls use of your content for Gemini training and grounding.
Do AI crawlers actually obey robots.txt?
The major vendors listed here say their automatic crawlers do. Several say their user-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) may not, because a person asked for the page. For a hard block, use your server or CDN.
Does blocking now remove content already collected?
No. robots.txt controls future crawling. Common Crawl notes existing snapshots are unchanged; ask vendors directly about data they already hold.