GPTBot: what it is and how to block it
Crawls content that may be used to train OpenAI’s generative AI foundation models.
Does GPTBot follow robots.txt?
OpenAI says disallowing GPTBot indicates your content should not be used for training. Each OpenAI token is independent.
What blocking it does
Opts future crawls out of model training. It does not remove you from ChatGPT search, which is controlled by OAI-SearchBot.
User-agent string
How it appears in your server logs (from OpenAI's documentation; the version number can change):
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Anyone can send this string. Verify real visits against the operator's published IP ranges or reverse DNS before trusting it.
Other OpenAI tokens
- OAI-SearchBot: ai search index. Surfaces websites in search results in ChatGPT’s search features.
- ChatGPT-User: user-requested fetch. Visits pages when a ChatGPT user or a Custom GPT asks for them. It doesn’t crawl automatically.
Block GPTBot everywhere
User-agent: GPTBot
Disallow: /
Block only some folders
User-agent: GPTBot
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: GPTBot
Allow: /
A group that names GPTBot replaces the * group for it. Copy any shared rules into this group too. Test the result.
GPTBot questions
What is GPTBot?
GPTBot is operated by OpenAI. Crawls content that may be used to train OpenAI’s generative AI foundation models.
How do I block GPTBot?
Add "User-agent: GPTBot" followed by "Disallow: /" to your robots.txt. OpenAI says disallowing GPTBot indicates your content should not be used for training. Each OpenAI token is independent.
What happens if I block GPTBot?
Opts future crawls out of model training. It does not remove you from ChatGPT search, which is controlled by OAI-SearchBot.
Does blocking GPTBot block other OpenAI crawlers?
No. OpenAI documents OAI-SearchBot, ChatGPT-User separately; each needs its own User-agent line if you want to block it.