RobotsTxt Pro

Noindex in robots.txt

Putting Noindex: in robots.txt does nothing on Google. It never was part of the standard, and Google stopped honouring it in 2019.

What happened to robots.txt noindex

Google announced in July 2019 that from 1 September 2019 it would stop supporting unpublished rules in robots.txt, noindex included, as it pushed the Robots Exclusion Protocol towards an internet standard. That standard became RFC 9309 in 2022, and it defines only user-agent, allow and disallow. Google also documents sitemap.

Disallow is not noindex

Disallow stops crawling. But a URL Google has never crawled can still be indexed from links alone, shown without a snippet. Worse, if you disallow a page that carries a noindex tag, Google can’t crawl it to see the tag.

What to use instead

Robots meta tag (HTML pages)

<meta name="robots" content="noindex">

X-Robots-Tag header (PDFs, images, any file)

X-Robots-Tag: noindex
  1. Remove any Disallow rule covering the page so crawlers can reach it.
  2. Add the meta tag or header.
  3. Wait for recrawl, or request indexing in Search Console to speed it up.
  4. Once it has dropped out, you can leave the tag in place. Don’t re-block it in robots.txt.

The validator flags Noindex, Nofollow and other unsupported fields in any robots.txt you paste.

Questions

Does Bing support noindex in robots.txt?

No. Neither Google nor Bing documents it. Use the robots meta tag or X-Robots-Tag header, which both support.

How do I noindex a whole folder?

Send an X-Robots-Tag: noindex header for every URL in the folder from your server or CDN config. Robots.txt can’t do it.

Sources: Google Search Central blog, July 2019: A note on unsupported rules in robots.txt; Google: Block indexing with noindex; RFC 9309.