Robots.txt Tester
Paste a robots.txt file's content and a URL path to test against it, and see exactly which rule (if any) allows or blocks that path for a given user-agent — no live crawl needed.
The test happens entirely on your device — the robots.txt content is never sent to a server.
What this tool does
Tests whether a given path is allowed or disallowed for a given crawler under the robots.txt content you paste, and shows which rule matched.
When to use it
- A page is missing from search and you suspect the crawler file blocks it.
- Checking a file before uploading it.
- Testing a complex rule involving wildcards.
- Confirming a block covers what you intend and nothing more.
How to use it
- Paste the "robots.txt content" into the field.
- Enter the "Crawler user-agent" to test as.
- Enter the "Path to test".
- Click "Test" and read the verdict and the "Matched rule".
A practical example
Input
A file disallowing /admin/ with the path /admin/login tested.
Output
A "Disallowed" verdict naming the rule that matched the path.
Limitations
- The test runs on the content you paste; it does not fetch your live file.
- Crawlers differ slightly in how they interpret some patterns, so a specific crawler's behaviour may differ in edge cases.
- The verdict is about crawling rather than indexing, which are different things.
Privacy and processing
The test happens entirely on your device and the robots.txt content is never sent to a server.
Technical notes
- Showing the matched rule is what makes this useful: in a file with conflicting rules you learn which one won and why, rather than just getting a yes or no.
Frequently Asked Questions
Yes — testing a URL against your robots.txt rules and checking whether it's allowed or blocked is the same task. Paste your robots.txt content and the URL path you want to check to see the exact matching rule.