Robots.txt — Definition & Meaning
What is Robots.txt?
robots.txt is a plain-text file at your domain root (yoursite.com/robots.txt) that tells crawlers which URLs they can and cannot request. It controls crawl budget, blocks staging paths, and points crawlers at your sitemap — but it does not prevent indexation.
Key points
- Location must be exactly /robots.txt at the root — subdirectory or subdomain files are ignored.
- Disallow blocks crawling, NOT indexing — a URL blocked in robots.txt can still be indexed if linked externally. Use noindex meta tags for indexation control.
- Add "Sitemap: https://yoursite.com/sitemap.xml" so crawlers discover your sitemap without waiting for GSC.
- One accidental "Disallow: /" wipes your site from search overnight — always test in GSC's robots.txt tester before deploying.
- Different crawlers respect robots.txt differently — Googlebot and Bingbot are strict; scrapers and LLM crawlers often ignore it.
Example
A staging deploy that forgot to remove `Disallow: /` from robots.txt can drop the production site from Google in hours. This is the single most common cause of overnight organic traffic loss.
Frequently asked questions
Does robots.txt prevent Google from indexing a page?
No — it prevents crawling. A URL blocked in robots.txt can still appear in results (usually with no snippet) if linked externally. Use a noindex meta tag to actually block indexation.
How do I test my robots.txt without breaking things?
Use the free Robots.txt Tester at /free-tools/robots-txt-tester — paste your file and any URL to check what's allowed vs blocked. Google Search Console also has a legacy tester under Settings.
Should I block AI crawlers in robots.txt?
Depends on strategy — blocking GPTBot, ClaudeBot, and PerplexityBot prevents your content from training LLMs but also removes you from LLM citations, which are now a growing traffic channel.