Claude-SearchBot: what it is and how to block it
Navigates the web to improve the quality of search results for Claude users.
Does Claude-SearchBot follow robots.txt?
Honours robots.txt, per Anthropic.
What blocking it does
Anthropic says blocking it stops your content being indexed for search, which may reduce your visibility in Claude’s search results.
Other Anthropic tokens
- ClaudeBot: model training. Collects web content that could contribute to training Anthropic’s generative AI models.
- Claude-User: user-requested fetch. Fetches pages when a Claude user asks a question that needs them.
Block Claude-SearchBot everywhere
User-agent: Claude-SearchBot
Disallow: /
Block only some folders
User-agent: Claude-SearchBot
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: Claude-SearchBot
Allow: /
A group that names Claude-SearchBot replaces the * group for it. Copy any shared rules into this group too. Test the result.
Claude-SearchBot questions
What is Claude-SearchBot?
Claude-SearchBot is operated by Anthropic. Navigates the web to improve the quality of search results for Claude users.
How do I block Claude-SearchBot?
Add "User-agent: Claude-SearchBot" followed by "Disallow: /" to your robots.txt. Honours robots.txt, per Anthropic.
What happens if I block Claude-SearchBot?
Anthropic says blocking it stops your content being indexed for search, which may reduce your visibility in Claude’s search results.
Does blocking Claude-SearchBot block other Anthropic crawlers?
No. Anthropic documents ClaudeBot, Claude-User separately; each needs its own User-agent line if you want to block it.