RobotsTxt Pro

How to create a robots.txt file

A robots.txt file is a plain text file with a handful of lines. Here’s how to write one, put it in the right place and prove it works.

1. Decide what crawlers should skip

List the areas that have no value in search: admin and login, cart and checkout, internal search results, filter and sort parameters, staging copies. Keep everything visitors should find crawlable.

2. Write the rules

# All crawlers
User-agent: *
Disallow: /admin/
Disallow: /search
Disallow: /*?sort=

Sitemap: https://example.com/sitemap.xml

Each group starts with one or more User-agent lines, then Disallow and Allow paths starting with /. The generator writes this for you, including from a plain-English description.

3. Save it correctly

  • Name it exactly robots.txt, all lower case.
  • Save as UTF-8 plain text. Word processors add formatting that breaks it; use a code editor or Notepad.
  • Lines can end in LF or CR/LF.

4. Upload it to the root

It must be reachable at https://yourdomain/robots.txt. A file at /blog/robots.txt is ignored. On WordPress, Shopify, Blogger and other hosted platforms, use the platform’s method; see the templates.

5. Test it

  • Open the URL in a browser and check that you see your text, not an HTML page.
  • Fetch it in the validator to check status code, content type and syntax.
  • Run your most important URLs through the tester as Googlebot to make sure they’re allowed.
  • In Google Search Console, the robots.txt report shows when Google last fetched it and any problems it found.

Questions

Can I create robots.txt without FTP access?

Usually. WordPress SEO plugins, Shopify’s robots.txt.liquid, Blogger settings and most hosting file managers let you create it from the browser.

Does robots.txt need to be in the root of every subdomain?

Yes. blog.example.com and example.com are separate hosts and each reads only its own /robots.txt.

Sources: Google: Create and submit a robots.txt file; RFC 9309 §2.3 Access method.