RobotsTxt Pro

WordPress robots.txt template

WordPress serves a virtual robots.txt until you upload a real file. The default is sensible; most sites only need to add a sitemap line, block internal search results and decide what to do about AI crawlers.

The template

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/wp-sitemap.xml

Replace example.com with your domain before publishing.

Where the file lives on WordPress

  • WordPress generates /robots.txt on the fly. Uploading a physical robots.txt to your site root (the folder with wp-config.php) replaces it.
  • SEO plugins such as Yoast SEO and Rank Math include a robots.txt editor, which writes that file for you.
  • Since WordPress 5.5, core publishes an XML sitemap at /wp-sitemap.xml. If an SEO plugin replaces it (Yoast uses /sitemap_index.xml), use that URL instead.

Why these rules

  • Keep Allow: /wp-admin/admin-ajax.php. Themes and plugins call it from the front end, and blocking it can break how Google renders pages.
  • Don’t block /wp-content/ or /wp-includes/. They hold the CSS, JavaScript and images Google needs to render your pages.
  • Blocking /?s= and /search/ keeps crawlers out of endless internal search pages.
  • Settings → Reading → “Discourage search engines” adds a noindex tag; it doesn’t edit robots.txt. Turn it off when a site goes live.

After publishing, fetch your live file in the validator to confirm WordPress serves what you expect.

WordPress robots.txt questions

Where is robots.txt in WordPress?

By default there is no file on disk: WordPress generates /robots.txt dynamically. Upload a robots.txt file to the site root, or use an SEO plugin’s file editor, to replace it.

Should I block /wp-content/ in robots.txt?

No. It contains your theme and plugin CSS and JavaScript plus uploaded images. Google needs those to render your pages, and blocking them can hurt how your pages are understood.

Does blocking /wp-admin/ hide my login page?

It only asks crawlers not to fetch it. robots.txt is public and is not a security control; protect the login with strong passwords and two-factor authentication.

Other platforms