User-agent: ClaudeBot
Crawl-delay: 5The value is usually read as seconds between requests. It belongs inside a group, after the User-agent lines.
Who reads it
- Google: no. Google’s robots.txt documentation lists crawl-delay among unsupported fields.
- Anthropic: yes. Anthropic’s crawler documentation says its bots support the non-standard Crawl-delay extension.
- Others: check each crawler’s own docs. If a vendor doesn’t document it, assume it’s ignored.
Slowing Googlebot down
- Google adjusts its crawl rate automatically based on how quickly your server responds.
- For an urgent slowdown, return
503or429for a short time. Google backs off. Don’t do it for more than a day or two, or pages may start to drop. - Don’t use 401 or 403 on robots.txt to limit crawling. Google treats 4xx robots.txt responses as “no restrictions”.
A high Crawl-delay on a crawler that honours it can slow the discovery of new pages a lot. At 10 seconds, a bot fetches no more than 8,640 URLs a day.
Questions
What is a good crawl-delay value?
Only set one if a specific bot is straining your server. Start at 1–2 seconds for that bot’s group rather than for "*".
Does crawl-delay hurt SEO?
Not on Google, which ignores it. For crawlers that honour it, a large value slows how fast they find new pages.
Sources: Google: robots.txt specification; Anthropic: Does Anthropic crawl data from the web?; Google: Reduce Googlebot crawl rate.