SEO & Search
What Is Crawl Budget?
The amount of crawling Google will do on a site, set by how much the server can take and how much Google wants the content.
Definition
Crawl budget is the number of URLs Googlebot can and wants to crawl on a site in a given period. Google describes it as the product of two factors: the crawl capacity limit, which is how much crawling the site can serve without degrading, and crawl demand, which is how much Google wants the content based on popularity, value and staleness.
Who needs to think about it
Google is explicit that most sites do not need to manage crawl budget. It becomes a practical concern for large sites, for sites that generate many URLs automatically, and for sites where content changes frequently enough that recrawl speed affects revenue.
The two levers that actually move it
Google states that the only ways to increase crawl budget are to increase serving capacity and to increase the value of the content to searchers. Most day to day work is therefore not about raising the budget but about not wasting it.
- Remove or consolidate duplicate and near duplicate URLs
- Block crawling of URL patterns that produce no useful index entries
- Fix redirect chains, which spend several requests to reach one document
- Return correct status codes, especially 404 and 410 for removed pages
- Keep server response times low, since slow responses reduce the capacity limit
How to observe it
The Crawl Stats report in Search Console shows request volume, response codes and average response time over time. Server logs give the same picture at URL level, which is where wasted crawling usually becomes obvious.
The only ways to increase crawl budget are to increase serving capacity and to increase the value of your content to searchers.