Robots.txt Templates

10 pre-built templates for common website types

📝

WordPress

Standard robots.txt for WordPress sites. Blocks admin areas and system files.

CMSBlogPopular

16 lines

robots.txt

# WordPress Robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow...

Use

🛍️

E-commerce

Perfect for online stores. Blocks checkout, cart, and user account pages.

E-commerceShoppingSecure

15 lines

robots.txt

# E-commerce Robots.txt
User-agent: *
Disallow: /checkout/
Disallow: /cart/
Disallow: /my-account/
Disallow: /order-tracking/
Disallow: /wishlist/
Disallow: /compare/
Disallow: /ad...

Use

✍️

Blog / Content

For blogs and content-focused sites. Allows full crawling of content.

BlogContentSEO-friendly

10 lines

robots.txt

# Blog / Content Site Robots.txt
User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /search/
Disallow: /author/*/page/
Disallow: /*?s=
Allow: /

Sitemap: https://example.c...

Use

🎨

Portfolio

For portfolio and agency websites. Maximum visibility for your work.

PortfolioAgencySimple

5 lines

robots.txt

# Portfolio / Agency Robots.txt
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml...

Use

🔒

Strict Mode

Only allows Google and Bing. Blocks all other bots for maximum control.

SecurityRestrictedPrivate

11 lines

robots.txt

# Strict - Block Most Bots
User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Sitemap: https://example.com/sitemap.xml...

Use

☁️

SaaS Platform

For SaaS applications. Blocks user dashboards and API endpoints.

SaaSAppProtected

15 lines

robots.txt

# SaaS Platform Robots.txt
User-agent: *
Disallow: /app/
Disallow: /dashboard/
Disallow: /api/
Disallow: /login/
Disallow: /signup/
Disallow: /account/
Disallow: /settings/
Allow: ...

Use

⚡

Next.js

Optimized for Next.js applications with SSR, API routes, and static assets.

ReactFrameworkModern

10 lines

robots.txt

# Next.js Robots.txt
User-agent: *
Disallow: /api/
Disallow: /_next/webpack-hmr
Disallow: /*.json$
Allow: /api/og/*
Allow: /_next/static/
Allow: /_next/image/

Sitemap: https://exa...

Use

🔧

Laravel

For Laravel PHP applications. Blocks admin routes, storage, and vendor files.

PHPFrameworkBackend

13 lines

robots.txt

# Laravel Robots.txt
User-agent: *
Disallow: /admin/
Disallow: /storage/
Disallow: /vendor/
Disallow: /config/
Disallow: /database/
Disallow: /resources/
Disallow: /*.blade.php$
Di...

Use

🐍

Django

Python Django framework template. Blocks admin panel, media uploads, and static files.

PythonFrameworkBackend

11 lines

robots.txt

# Django Robots.txt
User-agent: *
Disallow: /admin/
Disallow: /accounts/
Disallow: /api/
Disallow: /__debug__/
Disallow: /static/admin/
Allow: /static/
Allow: /media/

Sitemap: htt...

Use

💎

Ruby on Rails

Ruby on Rails application template. Blocks admin routes, assets compilation, and user accounts.

RubyFrameworkBackend

11 lines

robots.txt

# Ruby on Rails Robots.txt
User-agent: *
Disallow: /admin/
Disallow: /users/
Disallow: /api/
Disallow: /rails/
Disallow: /cable/
Allow: /assets/
Allow: /packs/

Sitemap: https://ex...

Use