ClaudeBot: what it is and how to block it
Collects web content that could contribute to training Anthropic’s generative AI models.
Does ClaudeBot follow robots.txt?
Anthropic says its bots honour robots.txt and also support the non-standard Crawl-delay extension. It warns that blocking its IPs instead may stop it reading your robots.txt.
What blocking it does
Signals that your site’s future content should be excluded from Anthropic’s training datasets.
Other Anthropic tokens
- Claude-SearchBot: ai search index. Navigates the web to improve the quality of search results for Claude users.
- Claude-User: user-requested fetch. Fetches pages when a Claude user asks a question that needs them.
Block ClaudeBot everywhere
User-agent: ClaudeBot
Disallow: /
Block only some folders
User-agent: ClaudeBot
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: ClaudeBot
Allow: /
A group that names ClaudeBot replaces the * group for it. Copy any shared rules into this group too. Test the result.
ClaudeBot questions
What is ClaudeBot?
ClaudeBot is operated by Anthropic. Collects web content that could contribute to training Anthropic’s generative AI models.
How do I block ClaudeBot?
Add "User-agent: ClaudeBot" followed by "Disallow: /" to your robots.txt. Anthropic says its bots honour robots.txt and also support the non-standard Crawl-delay extension. It warns that blocking its IPs instead may stop it reading your robots.txt.
What happens if I block ClaudeBot?
Signals that your site’s future content should be excluded from Anthropic’s training datasets.
Does blocking ClaudeBot block other Anthropic crawlers?
No. Anthropic documents Claude-SearchBot, Claude-User separately; each needs its own User-agent line if you want to block it.