CCBot: what it is and how to block it
Builds Common Crawl’s open web archive, which many AI training datasets are built from.
Does CCBot follow robots.txt?
Common Crawl documents "User-agent: CCBot / Disallow: /" as the opt-out and warns that some crawlers falsely identify as CCBot.
What blocking it does
Keeps new pages out of future Common Crawl snapshots. Existing snapshots are not changed.
User-agent string
How it appears in your server logs (from Common Crawl's documentation; the version number can change):
CCBot/2.0 (https://commoncrawl.org/faq/)
Anyone can send this string. Verify real visits against the operator's published IP ranges or reverse DNS before trusting it.
Block CCBot everywhere
User-agent: CCBot
Disallow: /
Block only some folders
User-agent: CCBot
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: CCBot
Allow: /
A group that names CCBot replaces the * group for it. Copy any shared rules into this group too. Test the result.
CCBot questions
What is CCBot?
CCBot is operated by Common Crawl. Builds Common Crawl’s open web archive, which many AI training datasets are built from.
How do I block CCBot?
Add "User-agent: CCBot" followed by "Disallow: /" to your robots.txt. Common Crawl documents "User-agent: CCBot / Disallow: /" as the opt-out and warns that some crawlers falsely identify as CCBot.
What happens if I block CCBot?
Keeps new pages out of future Common Crawl snapshots. Existing snapshots are not changed.