Custom robots.txt for Blogger
Blogger serves a sensible robots.txt by default. Turn on the custom file only if you need to change it, such as adding a sitemap for a custom domain or blocking AI crawlers.
The template
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://example.blogspot.com/sitemap.xml
Replace example.com with your domain before publishing.
Where the file lives on Blogger
- Blogger dashboard → Settings → Crawlers and indexing → turn on “Enable custom robots.txt” → click “Custom robots.txt” and paste.
- Use your real blog address in the Sitemap line, including a custom domain if you have one.
Why these rules
- Disallow: /search keeps label pages (/search/label/…) and search results out, which are thin duplicates of your posts.
- Mediapartners-Google is the AdSense crawler. An empty Disallow lets it read every page so ads can match your content.
- Leave “Custom robots header tags” alone unless you know you need them; those are meta robots tags, not robots.txt.
After publishing, fetch your live file in the validator to confirm Blogger serves what you expect.
Blogger robots.txt questions
Should I enable custom robots.txt on Blogger?
Only if you need to change the default. A wrong custom file can block your whole blog, so validate it before saving.
Why does Blogger block /search?
Label and search result pages repeat content from your posts. Keeping crawlers out of them focuses crawling on the posts themselves.