Free Robots.txt Tester — Check URL Access
Robots.txt Tester
A robots.txt tester tells you whether a specific crawler is allowed to fetch a specific URL, by applying the same matching rules Google uses: the most specific matching rule wins, and when an Allow and a Disallow are equally specific, Allow wins. Paste your robots.txt and a URL below to see the verdict and the exact line that produced it.
Paste your robots.txt, enter a URL, and instantly see whether Googlebot (or any crawler) is allowed to fetch it. Free client-side tester, no signup.
Paste the live robots.txt
Fetch it from yourdomain.com/robots.txt and paste the whole file, including every user-agent block. Rules in one block never apply to another agent.
Pick the crawler you care about
Googlebot obeys the most specific user-agent block that names it, falling back to the wildcard block. GPTBot, Bingbot and others resolve independently — a URL open to Google can be closed to LLM crawlers.
Read the matching rule, not just the verdict
The tester shows which line matched. Most robots.txt bugs are an over-broad Disallow that catches more paths than intended, not a missing rule.
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. Robots.txt blocks crawling, not indexing. A blocked URL with inbound links can still appear in Google with no description, because Google knows the URL exists but cannot read it. To keep a page out of the index, allow crawling and serve a noindex meta tag.
What is the difference between Disallow and noindex?
Disallow tells crawlers not to fetch the URL. Noindex tells them not to list it in search results. They conflict: if you disallow a page, Google never fetches it, never sees the noindex tag, and the page can linger in the index indefinitely.
Does robots.txt block ChatGPT and other AI crawlers?
Only if you name them. GPTBot, ClaudeBot, PerplexityBot and Google-Extended each read their own user-agent block. A wildcard Allow: / leaves your site open to all of them; blocking one does nothing to the others.
Where does robots.txt have to live?
At the root of the host — https://example.com/robots.txt. A file at /subfolder/robots.txt is ignored, and each subdomain needs its own file. Protocol and port also matter: http and https are separate hosts for robots purposes.