Claude-User: what it is and how to block it
Fetches pages when a Claude user asks a question that needs them.
Does Claude-User follow robots.txt?
Anthropic says Claude-User lets site owners control which sites these user-initiated requests can access.
What blocking it does
Claude won’t retrieve your pages in answer to user questions, which may reduce your visibility for user-directed web search.
Because Anthropic describes Claude-User as fetching pages on a user's behalf, treat a robots.txt rule as a request rather than a guarantee. For a hard block, filter the user agent or Anthropic's published IP ranges at your server or CDN.
Other Anthropic tokens
- ClaudeBot: model training. Collects web content that could contribute to training Anthropic’s generative AI models.
- Claude-SearchBot: ai search index. Navigates the web to improve the quality of search results for Claude users.
Block Claude-User everywhere
User-agent: Claude-User
Disallow: /
Block only some folders
User-agent: Claude-User
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: Claude-User
Allow: /
A group that names Claude-User replaces the * group for it. Copy any shared rules into this group too. Test the result.
Claude-User questions
What is Claude-User?
Claude-User is operated by Anthropic. Fetches pages when a Claude user asks a question that needs them.
How do I block Claude-User?
Add "User-agent: Claude-User" followed by "Disallow: /" to your robots.txt. Anthropic says Claude-User lets site owners control which sites these user-initiated requests can access.
What happens if I block Claude-User?
Claude won’t retrieve your pages in answer to user questions, which may reduce your visibility for user-directed web search.
Does blocking Claude-User block other Anthropic crawlers?
No. Anthropic documents ClaudeBot, Claude-SearchBot separately; each needs its own User-agent line if you want to block it.