RobotsTxt Pro

Robots.txt generator

Pick a starting point, tell the AI what you want in plain English, or tick rules yourself. Every line is checked against RFC 9309 and Google's documented rules as you type.

1. Start from

2. Describe it in plain English AI

Try:

The AI draft is checked by the same validator as everything else. Read it before you publish it.

3. Or pick rules yourself

All crawlers (User-agent: *)
AI crawlers to block

Tokens come from each vendor's own documentation. What each one does.

Google ignores this. Why

built from your options
# All other crawlers
User-agent: *
Disallow:
Test a URL

Live validation

✓ No errors or warnings3 notes
  • infoline 2 · Group for * has no rules, so those crawlers may crawl everything.
  • infoline 3 · Empty Disallow means "nothing is disallowed" for this group. That is valid and the rule itself is ignored.
  • infoline 4 · No Sitemap line. Adding "Sitemap: https://your-site/sitemap.xml" helps crawlers find your URLs.

Upload it as robots.txt to the root of your domain, e.g. https://example.com/robots.txt.

What goes into a good robots.txt

A robots.txt file is a list of groups. Each group names one or more crawlers with User-agent lines, then lists Allow and Disallow paths. A crawler reads only the group that names it most specifically, or the * group if nothing names it. Within that group, the longest matching path wins.

For most sites that means three decisions:

  1. What's not worth crawling. Admin screens, cart and checkout, internal search results, endless filter combinations. Blocking them saves crawl budget for pages that matter.
  2. Which AI crawlers to let in. Training crawlers and AI-search crawlers are separate tokens, so you can stay visible in ChatGPT or Perplexity search while opting out of model training.
  3. Where your sitemap lives. One Sitemap: line with a full URL.

What robots.txt can't do: hide pages from search results (use noindex), protect private content (use authentication), or force a bot to obey. It's a published request that well-behaved crawlers honour.

Robots.txt generator questions

Is this robots.txt generator free?

Yes. The generator, validator, URL tester and templates are free with no sign-up. The AI writer is free too, with a per-minute limit to keep it fair.

How does the AI robots.txt generator work?

You describe what crawlers should and shouldn’t reach in plain English. A language model drafts the rules, then the same RFC 9309 validator that checks hand-written files checks the draft and flags anything risky before you copy it.

Where do I put the robots.txt file?

At the root of the host it controls: https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain and protocol needs its own file (RFC 9309 §2.3).

How do I add my sitemap to robots.txt?

Add a line such as "Sitemap: https://example.com/sitemap.xml" anywhere in the file. It must be a full URL and it applies to all crawlers, whichever group it sits near.

Will robots.txt remove a page from Google?

No. Disallow stops crawling, but a blocked URL can still be indexed without a snippet if other pages link to it. To keep a page out of results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header.

Can I block ChatGPT, Claude and other AI bots?

Yes, for the crawlers that honour robots.txt. Each vendor publishes its own token (GPTBot, ClaudeBot, Google-Extended and so on). Tick them in step 3; the AI crawlers page explains what each one controls.

How long until changes take effect?

Google generally caches robots.txt for up to 24 hours. OpenAI says its search crawler takes about 24 hours to pick up changes, and Meta gives the same figure.