Robots.txt wildcards

Try a pattern

The paths below come from Google’s own wildcard table. Change the rule and watch which ones it catches.

How matching works

• Rules are compared from the start of the path, including the query string. /fish matches /fish.php?id=1 but not /catfish.
• * matches zero or more of any character, including / and ?. A trailing * changes nothing: /fish* equals /fish.
• $ is only special at the very end. /*.php$ matches /a.php but not /a.php?x=1.
• Paths are case-sensitive: /*.php does not match /windows.PHP.
• Non-ASCII characters are compared in percent-encoded form, so /café and /caf%C3%A9 are the same rule (RFC 9309 §2.2.2).

Wildcards and precedence

When Allow and Disallow both match, the longer rule path wins, and * counts as a character. That’s why Disallow: /*.htm (6 characters) beats Allow: /page (5) for /page.htm. Step through all six of Google’s cases in the tester.

Common mistakes

• Disallow: *.pdf: no leading slash. Write /*.pdf.
• Disallow: /*? blocks every URL with a query string, including paginated and tracked links you may want crawled.
• Disallow: /private*/ works, but /private alone already covers /private-files/ and /privately. Be sure that’s what you want.
• Expecting $ to work mid-path, as in /a$b. It’s taken literally there.

Sources: Google: URL matching based on path values; RFC 9309 §2.2.3 Special characters.