Google-CloudVertexBot: what it is and how to block it
Crawls sites whose owners asked for it while building Vertex AI Agents.
Does Google-CloudVertexBot follow robots.txt?
Honours robots.txt rules addressed to Google-CloudVertexBot.
What blocking it does
Google says it has no effect on Google Search or other products.
Because Google describes Google-CloudVertexBot as fetching pages on a user's behalf, treat a robots.txt rule as a request rather than a guarantee. For a hard block, filter the user agent or Google's published IP ranges at your server or CDN.
Other Google tokens
- Google-Extended: usage control token. A product token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
Block Google-CloudVertexBot everywhere
User-agent: Google-CloudVertexBot
Disallow: /
Block only some folders
User-agent: Google-CloudVertexBot
Disallow: /members/
Disallow: /drafts/
Explicitly allow it
User-agent: Google-CloudVertexBot
Allow: /
A group that names Google-CloudVertexBot replaces the * group for it. Copy any shared rules into this group too. Test the result.
Google-CloudVertexBot questions
What is Google-CloudVertexBot?
Google-CloudVertexBot is operated by Google. Crawls sites whose owners asked for it while building Vertex AI Agents.
How do I block Google-CloudVertexBot?
Add "User-agent: Google-CloudVertexBot" followed by "Disallow: /" to your robots.txt. Honours robots.txt rules addressed to Google-CloudVertexBot.
What happens if I block Google-CloudVertexBot?
Google says it has no effect on Google Search or other products.
Does blocking Google-CloudVertexBot block other Google crawlers?
No. Google documents Google-Extended separately; each needs its own User-agent line if you want to block it.