Google-Extended: what it is and how to block it
A product token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
Does Google-Extended follow robots.txt?
Crawling is done by existing Google user agents; Google-Extended is read from robots.txt purely as a control signal.
What blocking it does
Google says it does not affect a site’s inclusion or ranking in Google Search.
Other Google tokens
- Google-CloudVertexBot: user-requested fetch. Crawls sites whose owners asked for it while building Vertex AI Agents.
Block Google-Extended everywhere
User-agent: Google-Extended
Disallow: /
Block only some folders
User-agent: Google-Extended
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: Google-Extended
Allow: /
A group that names Google-Extended replaces the * group for it. Copy any shared rules into this group too. Test the result.
Google-Extended questions
What is Google-Extended?
Google-Extended is operated by Google. A product token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
How do I block Google-Extended?
Add "User-agent: Google-Extended" followed by "Disallow: /" to your robots.txt. Crawling is done by existing Google user agents; Google-Extended is read from robots.txt purely as a control signal.
What happens if I block Google-Extended?
Google says it does not affect a site’s inclusion or ranking in Google Search.
Does blocking Google-Extended block other Google crawlers?
No. Google documents Google-CloudVertexBot separately; each needs its own User-agent line if you want to block it.