WordPress

WordPress serves /robots.txt from the site root. Choose the approach that matches your hosting setup.

Method 1: Manual upload (cPanel, SFTP, SSH)

1. Create and download your robots.txt file from our generator.
2. Connect to your server with FileZilla, Cyberduck, WinSCP, or SSH using the credentials from your host.
3. Upload robots.txt to the same directory as wp-config.php (usually public_html or www).
4. Set permissions to 644 (readable by everyone) and purge any caching plugins/CDN.
5. Verify at https://yourdomain.com/robots.txt and submit to Google Search Console if needed.

Method 2: Yoast SEO / All in One SEO

1. Install and activate your preferred SEO plugin.
2. Yoast: Tools → File editor → Create robots.txt → Paste your rules → Save.
3. All in One SEO: Search Appearance → Advanced → Robots.txt Editor → Enable custom robots.txt → Paste → Save Changes.
4. Clear any page caching (WP Rocket, W3 Total Cache, Cloudflare) so the new file serves immediately.

Managed WordPress hosts

• WP Engine, Kinsta, Flywheel: use the built-in file manager or SSH to upload to /www/wp-content/../robots.txt.
• SiteGround / Bluehost: File Manager → public_html → Upload robots.txt.

Method 3: Theme snippet / WP-CLI (copy-ready)

Prefer to keep robots.txt versioned with your theme? Drop the snippet below into functions.php or a site-specific plugin—WordPress will render your custom rules dynamically and you never have to upload a file again.

add_filter( 'robots_txt', function ( $output, $public ) {
    if ( ! $public ) {
        return "User-agent: *\nDisallow: /";
    }

    return trim("
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml
");
}, 10, 2 );

Deploy via git/CI, or run wp theme refresh / wp plugin activate your-plugin on managed hosts to publish instantly.

Shopify

1. Log in to your Shopify admin
2. Go to Online Store → Themes → Actions → Edit code
3. Find robots.txt.liquid in the "Templates" folder
4. Edit the file with your custom robots.txt content
5. Save changes

⚠️ Note: Shopify automatically adds some rules. Your custom rules will be appended.

Copy-ready robots.txt.liquid starter

{% layout none %}
User-agent: *
Disallow: /checkout/
Disallow: /cart
Disallow: /admin
Disallow: /orders
Allow: /collections/

Sitemap: https://{{ shop.primary_domain.host }}/sitemap.xml

# Respect Shopify defaults
{% render 'robots', directives: 'default' %}

Save, then preview at https://yourstore.com/robots.txt. Use Theme Settings → App embeds to keep the snippet under version control.

Squarespace

1. In Squarespace, open Website → Settings → Developer Tools → robots.txt (Business and Commerce plans).
2. Toggle Custom robots.txt, paste your rules from RobotsTxt Pro, and click Save.
3. Squarespace publishes instantly—visit https://yourdomain.com/robots.txt to confirm.
4. Legacy 7.0 sites: enable Developer Mode → connect via SFTP → upload robots.txt to the root directory.
5. After edits, clear site cache (Settings → Advanced → Website Availability → Clear Cache) if changes do not appear right away.

ℹ️ Squarespace may serve a cached file for up to a few minutes—append ?v=1 during spot checks if needed.

Vercel

1. Place robots.txt in your project's public/ folder
2. Commit and push to your repository
3. Vercel will automatically deploy it
4. Accessible at yourdomain.com/robots.txt

Next.js (self‑hosted or Vercel)

Option 1: Static file

1. Add public/robots.txt to your Next.js project.
2. Commit the file and redeploy—Next.js automatically serves everything in public/ from the site root.
3. Verify locally at http://localhost:3000/robots.txt and again after deployment.

Option 2: Generate automatically with next-sitemap

1. Install: npm install next-sitemap --save-dev.
2. Create next-sitemap.config.js:

/** next-sitemap.config.js */
module.exports = {
  siteUrl: 'https://yourdomain.com',
  generateRobotsTxt: true,
  robotsTxtOptions: {
    policies: [
      { userAgent: '*', allow: '/' },
      { userAgent: 'GPTBot', disallow: ['/internal/'] },
    ],
  },
};

3. Add a script to package.json: \"postbuild\": \"next-sitemap\".
4. Run npm run build and deploy. The generated robots.txt will live in public/.

Option 3: App Router dynamic endpoint

Need different rules per environment? Export a typed config from app/robots.ts—Next.js serves it as plain text automatically.

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const isPreview = process.env.NEXT_PUBLIC_STAGE === 'preview'

  return {
    rules: [
      {
        userAgent: '*',
        allow: isPreview ? [] : ['/'],
        disallow: isPreview ? ['/'] : ['/api/internal', '/drafts'],
      },
      { userAgent: 'GPTBot', disallow: ['/customer-data'] },
    ],
    sitemap: [
      'https://example.com/sitemap.xml',
      'https://example.com/blog-sitemap.xml',
    ],
  }
}

Push to Vercel—no rewrites necessary and the endpoint stays version-controlled.

Netlify

1. Place robots.txt in your project's public/ or root folder
2. Deploy your site (git push or drag-and-drop)
3. File will be served at the root of your domain

Laravel

Option 1: Static file

1. Copy robots.txt into the Laravel public/ directory.
2. Commit and deploy; Laravel serves everything in public/ directly.
3. Confirm at https://yourdomain.com/robots.txt.

Option 2: Dynamic response

Useful when you need environment-specific rules.

// routes/web.php
Route::get('/robots.txt', function () {
    $content = view('robots')->render();
    return response($content, 200)
        ->header('Content-Type', 'text/plain');
});

Create resources/views/robots.blade.php with your rules and deploy.

// resources/views/robots.blade.php
User-agent: *
Disallow: /admin/
Disallow: /api/

@env('production')
Sitemap: https://example.com/sitemap.xml
@else
Disallow: /
@endenv

Commit the Blade view so staging/production differences stay in git, then run php artisan config:cache and redeploy.

Django

1. Add robots.txt to your static/ directory (or the app-level static folder).
2. Ensure django.contrib.staticfiles is enabled and STATICFILES_DIRS includes the path.
3. Collect static files: python manage.py collectstatic before deployment.

Serve via TemplateView (ideal for dynamic rules)

# urls.py
from django.urls import path
from django.views.generic import TemplateView

urlpatterns = [
    # ...
    path(
        "robots.txt",
        TemplateView.as_view(template_name="robots.txt", content_type="text/plain"),
    ),
]

Ruby on Rails

1. Place robots.txt in the Rails public/ directory and redeploy.
2. Assets in public/ bypass the asset pipeline and are served directly.

Need environment-specific rules?

# config/routes.rb
Rails.application.routes.draw do
  # ...
  get '/robots.txt' => 'robots#index'
end

# app/controllers/robots_controller.rb
class RobotsController < ApplicationController
  layout false

  def index
    rules = Rails.env.production? ? Rails.root.join('config/robots.prod.txt') : Rails.root.join('config/robots.dev.txt')
    render plain: File.read(rules), content_type: 'text/plain'
  end
end

GitHub Pages

1. Add robots.txt to your repository root
2. Commit and push to your gh-pages branch
3. GitHub will serve it automatically

Apache Server

1. Upload robots.txt to your public_html or htdocs folder
2. Ensure file permissions are set to 644 (readable by all)
3. No .htaccess configuration needed

Nginx Server

1. Place robots.txt in your site's root directory (e.g., /var/www/html)
2. Ensure proper permissions: chmod 644 robots.txt
3. Nginx serves static files by default