Robots.txt URL Tester
Test if specific URLs are blocked or allowed by your robots.txt rules. Test individual URLs or bulk test multiple URLs at once.
Your Robots.txt Rules
Paste your robots.txt content here to test URLs against it.
Test Single URL
Bulk URL Testing
Click to upload or drag and drop
CSV or TXT files with URLs
CSV format: one URL per line or "URL" column in first column •
🔍 How It Works
This tool simulates how search engine bots interpret your robots.txt file when accessing specific URLs.
- •Allow rules override Disallow rules
- •Specific paths take precedence over wildcards
- •User-agent: * applies to all bots unless more specific rule exists
💡 Pro Tips
- •Test important pages before deploying robots.txt
- •Bulk test common URL patterns to catch mistakes
- •Test with different user-agents to ensure correct behavior
- •Remember: robots.txt is a suggestion, not enforcement
🔗 Share Configuration
Share your robots.txt configuration and test results with others via a URL.
⌨️ Keyboard Shortcuts
- Ctrl/Cmd + EnterTest single URL
- Ctrl/Cmd + Shift + EnterTest bulk URLs
- Ctrl/Cmd + KShare configuration