Robots.txt tester
Is this URL blocked for this crawler? Paste the file, enter a URL, pick a bot. You'll see the group it obeys, every rule that matches and the one that decides.
Need the live file? Fetch it in the validator, then paste it here.
No rule in the applicable group matches this URL, so it is allowed.
- Path tested
- /checkout/help/returns
- Group used
- User-agent: googlebot
- Matching rules
- none
Same URL, every crawler
- Googlebotallowed
- Googlebot-Imageallowed
- Googlebot-Newsallowed
- Storebot-Googleallowed
- Bingbotallowed
- GPTBotblocked
- OAI-SearchBotallowed
- ChatGPT-Userallowed
- ClaudeBotallowed
- Claude-SearchBotallowed
- Claude-Userallowed
- Google-Extendedallowed
- Google-CloudVertexBotallowed
- PerplexityBotallowed
- Perplexity-Userallowed
- Applebot-Extendedallowed
- CCBotallowed
- Meta-ExternalAgentallowed
- Meta-ExternalFetcherallowed
- Amazonbotallowed
- Amzn-SearchBotallowed
- Amzn-Userallowed
Which rule wins? Step through Google's own examples
Google's documentation lists six conflicts between Allow and Disallow. The rule: longest match wins, and a tie goes to Allow. Step through them to see the character count decide.
Both match. "/p" is 2 characters, "/" is 1, so the longer Allow wins.
Examples from Google Search Central: order of precedence for rules. The tester above runs the same matcher, which passes all six cases plus Google's full wildcard table in our test suite.
Tester questions
How do I test if a URL is blocked by robots.txt?
Paste the robots.txt, enter the URL, pick the crawler and read the verdict. The tester picks the group that crawler would obey, lists every rule that matches, and shows which one wins.
Which rule wins when Allow and Disallow both match?
The longest one, counted in characters of the rule path. If they are exactly as long, Google (and RFC 9309) picks Allow, the less restrictive rule. Order in the file doesn’t matter.
Why is Googlebot ignoring my User-agent: * rules?
Because the file has a group that names Googlebot. A crawler obeys only its most specific group; the * rules are not added to it. Repeat any shared rules inside the Googlebot group.
Is matching case-sensitive?
Paths are: Disallow: /Private does not block /private. User-agent names and field names are not.
Does the tester check query strings?
Yes. Rules are matched against the path plus the query string, so Disallow: /*?sort= blocks /shoes?sort=price.