Does robots.txt stop a page from being indexed?

No. Robots.txt blocks crawling, not indexing. A blocked URL with inbound links can still appear in Google with no description, because Google knows the URL exists but cannot read it. To keep a page out of the index, allow crawling and serve a noindex meta tag.

What is the difference between Disallow and noindex?

Disallow tells crawlers not to fetch the URL. Noindex tells them not to list it in search results. They conflict: if you disallow a page, Google never fetches it, never sees the noindex tag, and the page can linger in the index indefinitely.

Does robots.txt block ChatGPT and other AI crawlers?

Only if you name them. GPTBot, ClaudeBot, PerplexityBot and Google-Extended each read their own user-agent block. A wildcard Allow: / leaves your site open to all of them; blocking one does nothing to the others.

Where does robots.txt have to live?

At the root of the host — https://example.com/robots.txt. A file at /subfolder/robots.txt is ignored, and each subdomain needs its own file. Protocol and port also matter: http and https are separate hosts for robots purposes.

Free tool · No signup

Robots.txt Tester

Instantly check whether a URL is allowed or blocked for any user-agent — pure client-side, no data leaves your browser.

robots.txt

User-agent

Path to test

Allowed

Matched rule: allow: /

In short

A robots.txt tester tells you whether a specific crawler is allowed to fetch a specific URL, by applying the same matching rules Google uses: the most specific matching rule wins, and when an Allow and a Disallow are equally specific, Allow wins. Paste your robots.txt and a URL below to see the verdict and the exact line that produced it.

Worked example

What the output looks like

Input: robots.txt: "User-agent: * Disallow: /search Allow: /search/help" · URL tested: /search/help/filters
Output: Allowed for Googlebot — matched by Allow: /search/help
How to read it: Both rules match this URL, but Allow: /search/help is longer and therefore more specific, so it wins. Change the rule to Disallow: /search/ and the help subtree would still be reachable — the trailing slash only changes which paths match, not the precedence.

How it works

Paste the live robots.txt

Fetch it from yourdomain.com/robots.txt and paste the whole file, including every user-agent block. Rules in one block never apply to another agent.

Pick the crawler you care about

Googlebot obeys the most specific user-agent block that names it, falling back to the wildcard block. GPTBot, Bingbot and others resolve independently — a URL open to Google can be closed to LLM crawlers.

Read the matching rule, not just the verdict

The tester shows which line matched. Most robots.txt bugs are an over-broad Disallow that catches more paths than intended, not a missing rule.

What this tool does

Supports Allow / Disallow / wildcards
Per user-agent evaluation (Googlebot, Bingbot, GPTBot, custom)
Highlights the exact rule that matched
Zero server calls — runs entirely in your browser

Questions

Frequently asked

More free SEO tools

Learn the concept

Do it inside LemRank

What Is a Noindex Tag?

Crawl your whole site and find every blocked URL

LemRank tracks rankings, backlinks, and AI-search visibility in one dashboard. 14-day free Pro trial. No credit card.