How to Use the Robots.txt Tester
How to Use the Robots.txt Tester
Free tool that checks whether robots.txt is blocking Googlebot (or any user-agent) from a specific URL.
Paste any URL to see whether robots.txt blocks Googlebot from crawling it — free, no signup — LemRank.
Frequently asked questions
Does the robots.txt tester execute JavaScript?
No — robots.txt is a plain text file at /robots.txt. The tester fetches it directly and evaluates the rules for the user-agent you select.
Which user-agent should I test?
Googlebot for search indexing. Googlebot-Image for image search. Bingbot for Bing. Use the specific bot most relevant to the traffic you're optimising for.
Why is my page blocked even though robots.txt allows it?
Check for meta robots noindex tags, X-Robots-Tag HTTP headers, and authentication walls. Robots.txt is only one of several ways a page can be blocked from indexing.