RobotsTxt Pro

Adding a sitemap to robots.txt

One line tells every crawler where your sitemap is. The rules are strict about the URL format and relaxed about everything else.

Sitemap: https://example.com/sitemap.xml

The rules

  • Use a full absolute URL, including https:// and the host. Sitemap: /sitemap.xml is invalid and ignored.
  • It isn’t tied to a group. Google’s docs say the sitemap field applies to all crawlers wherever it appears. Most people put it at the end.
  • List as many as you want. Google says there’s no limit on the number of Sitemap lines.
  • It can be on another host. The sitemap doesn’t have to live on the same host as robots.txt, for example a CDN.
  • The field name is case-insensitive; the URL is case-sensitive.

Multiple sitemaps or a sitemap index

Sitemap: https://example.com/sitemap_index.xml
Sitemap: https://example.com/news-sitemap.xml
Sitemap: https://cdn.example.org/image-sitemap.xml

If you have a sitemap index, listing just the index is enough. Crawlers find the child sitemaps through it.

Common platform sitemap URLs

  • WordPress core: /wp-sitemap.xml. Yoast SEO: /sitemap_index.xml.
  • Shopify: /sitemap.xml, added to robots.txt automatically.
  • Blogger: /sitemap.xml.

A Sitemap line helps crawlers discover URLs. It doesn’t override Disallow: if a URL in your sitemap is blocked, it still won’t be crawled. The tester shows conflicts like that.

Questions

Is a sitemap in robots.txt required?

No, but it’s free discovery for every crawler that reads robots.txt, including ones you can’t submit a sitemap to directly.

Should I still submit my sitemap in Search Console?

Yes. Submitting it gives you indexing reports for those URLs. The robots.txt line covers other crawlers.

Sources: Google: robots.txt specification (sitemap field); sitemaps.org protocol.