What is Crawl Budget?
Crawl budget is the number of URLs Googlebot will request on your site in a given period. It's a function of your site's host load capacity and Google's "crawl demand" (how important + fresh your content appears). Small sites rarely hit the limit; sites over ~10k URLs must manage it actively.
How Crawl Budget works
- Wasted budget: crawling parameter URLs, faceted navigation, session IDs, calendar rabbit-holes, and 404 chains.
- Reclaim budget with: robots.txt disallows on parameter patterns, noindex on thin variants, correct canonicals, and aggressive 301 consolidation.
- Freshness signals (real lastmod, frequent publishing, healthy backlinks) grow crawl demand — Google then allocates more budget.
- Watch "Crawl stats" in GSC — sudden drops signal a technical block; sudden spikes signal a duplicate-URL explosion.
- For under-10k-URL sites, focus on content quality first — budget is only a real constraint at ecommerce or news-site scale.
Example
An ecommerce site with 5,000 products but 500,000 faceted URLs (color × size × price combinations) burns 90% of Googlebot's time on duplicate junk — starving the real product pages of crawl attention.
Frequently asked questions about Crawl Budget
How do I check my crawl budget?
GSC → Settings → Crawl stats shows requests per day, response codes, and file types. Divide daily requests by total indexable URLs to get your effective crawl rate.
Does blocking a page in robots.txt save crawl budget?
Yes — Googlebot skips disallowed URLs entirely. This is the fastest way to reclaim budget from parameter and faceted URLs.
Do small sites need to worry about crawl budget?
Rarely. Under ~10k indexable URLs, Google will happily crawl the whole site every few days. Focus on content quality and internal linking instead.
Related terms
Robots.txt
robots.txt is a plain-text file at your domain root (yoursite.com/robots.txt) that tells crawlers which URLs they can and cannot request. It controls crawl budget, blocks staging paths, and points crawlers at your sitemap — but it does not prevent indexation.
XML Sitemap
An XML sitemap is a machine-readable list of the URLs on your site that you want search engines to crawl and index. Submitted via Google Search Console (or discovered through robots.txt), it accelerates discovery of new pages and communicates freshness with per-URL lastmod dates.
Indexation
Indexation is the process of Google adding your page to its searchable index. A page must be crawled, deemed valuable, and pass duplicate-content checks before it's indexed. GSC's Coverage report shows every indexation state — indexed, discovered-not-indexed, crawled-not-indexed, and excluded.
Noindex Tag
A noindex meta tag or X-Robots-Tag HTTP header tells search engines to crawl a page but exclude it from their index. It's the correct way to remove pages from Google — unlike robots.txt (which blocks crawling but not indexation of already-known URLs).