RobotsTxt Pro

Next.js robots.txt template

In the Next.js App Router you can put a static robots.txt in app/ or generate one with app/robots.ts. The TypeScript version is handy when rules depend on the environment, such as blocking everything on preview deployments.

The template

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Replace example.com with your domain before publishing.

app/robots.ts

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const isProd = process.env.VERCEL_ENV === 'production'
  if (!isProd) {
    return { rules: { userAgent: '*', disallow: '/' } }
  }
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/api/', '/admin/'] },
      { userAgent: 'GPTBot', disallow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

Where the file lives on Next.js

  • App Router: app/robots.txt (static) or app/robots.ts (generated). Next.js serves either at /robots.txt.
  • Pages Router: put a static robots.txt in public/.
  • Robots.ts is a special route handler that is cached by default unless it uses dynamic APIs.

Why these rules

  • Disallowing everything on preview deployments keeps staging URLs out of crawlers. Pair it with password protection if the content is private.
  • Don’t disallow /_next/. It serves the JavaScript and CSS Google needs to render your pages.

After publishing, fetch your live file in the validator to confirm Next.js serves what you expect.

Next.js robots.txt questions

Where does robots.txt go in Next.js?

App Router: app/robots.txt or app/robots.ts. Pages Router: public/robots.txt. All are served at /robots.txt.

Should I block /_next/ in robots.txt?

No. Those files are the JavaScript and CSS your pages need; blocking them stops Google rendering your pages properly.

Other platforms