What is Robots.txt?
robots.txt is a plain-text file at your domain root (yoursite.com/robots.txt) that tells crawlers which URLs they can and cannot request. It controls crawl budget, blocks staging paths, and points crawlers at your sitemap — but it does not prevent indexation.
How Robots.txt works
- Location must be exactly /robots.txt at the root — subdirectory or subdomain files are ignored.
- Disallow blocks crawling, NOT indexing — a URL blocked in robots.txt can still be indexed if linked externally. Use noindex meta tags for indexation control.
- Add "Sitemap: https://yoursite.com/sitemap.xml" so crawlers discover your sitemap without waiting for GSC.
- One accidental "Disallow: /" wipes your site from search overnight — always test in GSC's robots.txt tester before deploying.
- Different crawlers respect robots.txt differently — Googlebot and Bingbot are strict; scrapers and LLM crawlers often ignore it.
Example
A staging deploy that forgot to remove `Disallow: /` from robots.txt can drop the production site from Google in hours. This is the single most common cause of overnight organic traffic loss.
Frequently asked questions about Robots.txt
Does robots.txt prevent Google from indexing a page?
No — it prevents crawling. A URL blocked in robots.txt can still appear in results (usually with no snippet) if linked externally. Use a noindex meta tag to actually block indexation.
How do I test my robots.txt without breaking things?
Use the free Robots.txt Tester at /free-tools/robots-txt-tester — paste your file and any URL to check what's allowed vs blocked. Google Search Console also has a legacy tester under Settings.
Should I block AI crawlers in robots.txt?
Depends on strategy — blocking GPTBot, ClaudeBot, and PerplexityBot prevents your content from training LLMs but also removes you from LLM citations, which are now a growing traffic channel.
Related terms
XML Sitemap
An XML sitemap is a machine-readable list of the URLs on your site that you want search engines to crawl and index. Submitted via Google Search Console (or discovered through robots.txt), it accelerates discovery of new pages and communicates freshness with per-URL lastmod dates.
Crawl Budget
Crawl budget is the number of URLs Googlebot will request on your site in a given period. It's a function of your site's host load capacity and Google's "crawl demand" (how important + fresh your content appears). Small sites rarely hit the limit; sites over ~10k URLs must manage it actively.
Noindex Tag
A noindex meta tag or X-Robots-Tag HTTP header tells search engines to crawl a page but exclude it from their index. It's the correct way to remove pages from Google — unlike robots.txt (which blocks crawling but not indexation of already-known URLs).
Canonical URL
A canonical URL is the "master" version of a page you tell Google to index when the same or similar content is reachable from multiple URLs. Declared via <link rel="canonical" href="…" /> in the <head>, it consolidates ranking signals and prevents duplicate-content dilution.