RobotsTxt Pro

Robots.txt disallow all

"Disallow: /" asks every crawler to stay out of your entire site. It’s the right call for staging servers and the wrong one almost everywhere else.

Disallow all

User-agent: *
Disallow: /

The path / is a prefix of every URL, so every URL matches. Crawlers that have their own group elsewhere in the file (say, a User-agent: Googlebot group) ignore this one and follow theirs.

Allow all

User-agent: *
Disallow:

An empty Disallow is ignored, so nothing is blocked. Allow: / does the same thing. Having no robots.txt at all (a 404) is also treated as “allow everything” by Google and RFC 9309.

Blocked pages can still show up in Google

Disallow controls crawling, not indexing. If other sites link to a blocked URL, Google can still list it, usually without a description. Google says this in its own documentation. To keep a page out of results, let Google crawl it and add <meta name="robots" content="noindex">.

Staging sites: do this instead

  • Put staging behind HTTP authentication or an IP allow-list. Crawlers can’t index what they can’t load, and your unreleased content stays private.
  • If you do use Disallow: / on staging, make sure the deploy to production replaces it. Forgetting that is one of the most common ways sites vanish from search after a launch.
  • Frameworks let you switch rules by environment. See the Next.js template for an example.

Block everyone except one crawler

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

Googlebot matches its own group and ignores the * group. Everyone else is blocked.

Questions

How long does it take Google to notice Disallow: /?

Google generally caches robots.txt for up to 24 hours, so changes usually apply within a day.

Will Disallow: / remove my site from Google?

Not directly. Pages stop being crawled and their snippets disappear over time, but URLs can remain indexed if they are linked. Use noindex or the Search Console removals tool for fast removal.

Sources: Google: robots.txt specification; RFC 9309.