OAI-SearchBot: what it is and how to block it
Surfaces websites in search results in ChatGPT’s search features.
Does OAI-SearchBot follow robots.txt?
OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search, and notes robots.txt changes can take about 24 hours to apply.
What blocking it does
Your pages won’t be shown in ChatGPT search answers, though OpenAI says they can still appear as navigational links.
Other OpenAI tokens
- GPTBot: model training. Crawls content that may be used to train OpenAI’s generative AI foundation models.
- ChatGPT-User: user-requested fetch. Visits pages when a ChatGPT user or a Custom GPT asks for them. It doesn’t crawl automatically.
Block OAI-SearchBot everywhere
User-agent: OAI-SearchBot
Disallow: /
Block only some folders
User-agent: OAI-SearchBot
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: OAI-SearchBot
Allow: /
A group that names OAI-SearchBot replaces the * group for it. Copy any shared rules into this group too. Test the result.
OAI-SearchBot questions
What is OAI-SearchBot?
OAI-SearchBot is operated by OpenAI. Surfaces websites in search results in ChatGPT’s search features.
How do I block OAI-SearchBot?
Add "User-agent: OAI-SearchBot" followed by "Disallow: /" to your robots.txt. OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search, and notes robots.txt changes can take about 24 hours to apply.
What happens if I block OAI-SearchBot?
Your pages won’t be shown in ChatGPT search answers, though OpenAI says they can still appear as navigational links.
Does blocking OAI-SearchBot block other OpenAI crawlers?
No. OpenAI documents GPTBot, ChatGPT-User separately; each needs its own User-agent line if you want to block it.