Robots.txt validator
Fetch a live file or paste one in. You get a line-by-line syntax check, crawl-impact warnings, AI crawler coverage and an optional plain-English review.
Report
Paste or fetch a file to see the report.
What a crawler does with each HTTP status
The status code of /robots.txt matters as much as its content. A missing file opens the whole site; a broken server can close it. Pick a status to see how Google and RFC 9309 handle it.
Crawler follows your rules
- Rules in the file are applied.
- RFC 9309
- Parse and follow (§2.3.1.1).
Sources: Google, How Google interprets the robots.txt specification; RFC 9309 §2.3.1.
How the checks work
The parser reads the file the way RFC 9309 describes and Google documents: it splits on CR, LF or CR/LF, drops a leading byte-order mark and comments, lower-cases field names but keeps paths case-sensitive, and groups consecutive User-agent lines. It reads only the first 500 KiB, which is Google's limit.
Anything Google's own open-source parser tolerates but other crawlers might not, such as Disalow or a missing colon, is a warning rather than an error. Fields Google ignores (Crawl-delay, Host, Noindex) are flagged with what to use instead.
Validator questions
What does the robots.txt validator check?
Syntax (field: value lines, known fields, misspellings Google tolerates), grouping mistakes such as rules before any User-agent, paths that can’t match, relative sitemap URLs, the 500 KiB limit, HTML served instead of text, and crawl-impact risks like blocking the whole site or your CSS and JavaScript.
Is this the same as Google’s robots.txt tester?
Google retired the old robots.txt Tester in Search Console in 2023 and replaced it with a robots.txt report that shows fetch status. This tool follows the same published rules (RFC 9309 plus Google’s documentation), and our URL tester shows which rule decides for a given crawler and URL.
Why does my robots.txt return HTML?
Usually because /robots.txt doesn’t exist and the server returns a styled error or home page with a 200 status. Google tries to extract rules from it and ignores the rest. Serve a real text file instead.
What happens if robots.txt returns a 404?
Google treats any 4xx except 429 as “no robots.txt”: everything may be crawled. A 5xx or timeout is different: Google stops crawling for up to 12 hours, then falls back to its last cached copy.
Do you store the robots.txt files I check?
No. Pasted files are checked in your browser. Fetched files pass through our server only to get around browser cross-origin limits and aren’t stored. The AI review sends the file to the model only when you press the button.