Robots.txt templates by platform
Each platform stores and serves robots.txt differently. These templates start from the platform's sensible defaults and explain what to change.
- WordPress robots.txt templateA safe WordPress robots.txt: keep /wp-admin/ out, leave admin-ajax.php and your CSS/JS open, and point crawlers at wp-sitemap.xml. Copy it or customise it.
- Shopify robots.txt templateHow Shopify’s robots.txt works, what it already blocks, and a robots.txt.liquid template that keeps Shopify’s defaults while adding your own rules.
- Custom robots.txt for BloggerGenerate a custom robots.txt for Blogger/Blogspot: keep label and search pages out, keep posts crawlable, add your sitemap. Plus where to paste it in Blogger settings.
- Next.js robots.txt templateCreate robots.txt in Next.js with app/robots.ts (MetadataRoute.Robots) or a static public/robots.txt, including sitemap and AI crawler rules.
- Generic examplesAllow all, block all, block a folder, one bot, parameters, file types.