Key takeaways

  • Crawl budget is the set of URLs Google can and wants to crawl on your site. Google says it depends on two things: crawl capacity limit and crawl demand.
  • Most sites don’t need to worry about it. Google’s guide is written for sites with roughly 1 million+ pages, or 10,000+ pages that change daily, or many URLs stuck as “Discovered – currently not indexed”.
  • Server errors and slow responses lower how much Google crawls; useful, popular, changing pages raise how much it wants to crawl.
  • The fixes are mostly about waste: manage your URL inventory, consolidate duplicates, block useless URL spaces, return 404 or 410 for removed pages, fix soft 404s and redirect chains.
  • Read Crawl Stats by response type and purpose, and the Page indexing report by sitemap, before and after each fix.

Crawl budget is one of the most discussed and most misunderstood topics in technical SEO. On most sites it isn’t a problem at all. On large sites with deep structures, it can decide whether new products, listings or articles get crawled within days or sit undiscovered for weeks. The difference is size, and how many of your URLs are worth crawling.

I have spent a lot of my 16+ years in SEO on crawling problems for large sites. This article sticks closely to Google’s own guide: what crawl budget is, who should care, how to read the data, and the fixes that actually free up crawling.

What crawl budget is

Google’s large site owner’s guide to managing crawl budget says a site’s crawl budget “is determined by two main elements: crawl capacity limit and crawl demand.”

Element What it means What changes it
Crawl capacity limit How much crawling your server can take: Google says it “limits the total amount of time your server spends holding connections open for Google” Fast, healthy responses raise it; server errors (5xx) or rate-limiting signals such as 429 lower it
Crawl demand How much Google wants to crawl your URLs Perceived inventory, popularity and staleness

On demand, the guide lists three factors. Perceived inventory: Google “tries to crawl all or most of the URLs that it knows about on your site.” Popularity: “URLs that are more popular on the Internet tend to be crawled more often.” Staleness: “Our systems want to recrawl documents frequently enough to pick up any changes.”

The first factor is the one site owners control most directly. Every useless URL you expose adds to the inventory Google tries to crawl.

A gauge with two halves, one for server capacity and one for crawl demand, feeding a stream of page cards towards a search crawler path
Figure 1. Crawl budget comes from two sides: how much your server can take, and how much Google wants to crawl.

Does crawl budget matter for your site?

Probably not, unless your site is large or changes very quickly. Google’s guide is written for:

  • “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”;
  • “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”;
  • “Sites with a large portion of their total URLs classified by Search Console as Discovered – currently not indexed”.

Google calls these a rough estimate. For smaller sites, it says keeping your sitemap up to date and checking the Page indexing report regularly is enough. The Crawl Stats report makes the same point from the other side: “If you have a site with fewer than a thousand pages, you should not need to use this report.”

If your site has a few hundred pages and some aren’t indexed, crawl budget is almost never the reason.

On smaller sites, pages that aren’t indexed usually have a quality, duplication or linking problem, not a crawling one. Fixing “crawl budget” there is solving the wrong problem.

Rank Check Reports

If new pages seem slow to rank, check where your key keywords actually rank on desktop and mobile before you blame crawling.

A live Google check of your keywords on desktop and mobile · from $14.99 per check · no subscription · report within one working day

Order a rank check report →

How to see where your crawl budget goes

Crawl Stats in Search Console

The Crawl Stats report “is aimed at advanced users”, and on large sites it is the first place I look. It groups Googlebot’s requests by:

  • Response: how many requests got 200, 301, 404, 5xx and other codes;
  • File type: HTML, images, scripts, other resources;
  • Purpose: discovery (URLs never crawled before) or refresh (recrawls);
  • Googlebot type: which Googlebot made the request.

It also shows host status, which “describes whether or not Google encountered availability issues when trying to crawl your site.” If host status shows problems, fix them first: Google’s guide is clear that server errors lower the crawl capacity limit.

Server logs

Logs show every request, including the exact URLs. Grouped by URL pattern (filters, parameters, internal search, pagination), they show where crawling goes and how much of it reaches the pages you care about.

Page indexing by sitemap

The Page indexing report limits its example URLs to 1,000, so on a large site submit separate sitemaps per page type and read each one. A page type with many “Discovered – currently not indexed” URLs is where crawling isn’t reaching.

A pie of crawl requests split into a small gold slice for important pages and large grey slices for filter, parameter and redirect URLs, next to a log file with rows
Figure 2. Logs and Crawl Stats show where crawling goes; on many large sites, most of it lands on URLs nobody needs.

What crawl budget won’t fix

Crawl budget work gets pages crawled. It doesn’t make them rank. Three situations come up again and again where crawling is blamed for something else:

  • Pages are crawled but not indexed. “Crawled – currently not indexed” means Google fetched the page and chose not to index it. That points at quality, duplication or usefulness, not crawling.
  • Pages are indexed but don’t rank. That is a content, competition or links question. More crawling won’t change it.
  • A small site has a few unindexed pages. As Google’s guide says, smaller sites rarely need crawl budget work at all. Look at duplicates, thin pages and internal links instead.

Before starting crawl budget work, I check which of these is really happening, because each one needs a different fix and a different team. The Page indexing report usually answers it in a few minutes: “Discovered” problems are about crawling, “Crawled” problems are about the page.

How to fix crawl budget problems

Google’s guide is mostly about removing waste, so the fixes are too.

Crawl budget fixes, in the order I usually apply them

  • Fix host problems: server errors, timeouts and rate limiting lower how much Google crawls.
  • Manage your URL inventory: stop generating links to URL spaces that shouldn’t be crawled (filter combinations, sort orders, session or tracking parameters).
  • Consolidate duplicate content so one URL represents each page.
  • Block crawling of useless URL spaces with robots.txt (not noindex, which Google still requests).
  • Return a 404 or 410 for permanently removed pages.
  • Eliminate soft 404s: empty pages that return 200.
  • Keep sitemaps up to date with only canonical, indexable URLs.
  • Avoid long redirect chains; link straight to final URLs.
  • Make pages efficient to load.

A few notes from practice:

  • Noindex isn’t a crawl fix. Google says that for noindexed pages it “will still request, but then drop the page”. Use robots.txt or stop linking to them if they shouldn’t be crawled.
  • Faceted navigation is the most common cause on stores and listing sites. I covered it in detail in what an ecommerce SEO consultant fixes first.
  • Redirect chains build up after migrations. Update internal links to final URLs rather than relying on hops.

Can you increase crawl budget?

Google’s guide says “There are two ways to increase crawl budget”: “Add more server resources”, and improve the content itself, because Google allocates crawling based on factors such as popularity, user value, uniqueness and serving capacity. There is no setting to ask for more. In practice, removing waste usually helps faster than trying to raise the total.

Free tool · Ping My URL

After you fix crawl problems on key templates, let search engines know about the most important URLs that changed.

Notify Google and other search engines about up to 5 new or updated URLs at once. Free, no sign-up.

Ping your URLs →

Crawl budget on large sites: a short worked example

Picture a retailer with a few hundred thousand product and category URLs, plus filters that can combine colour, size, brand and price. Crawl Stats shows most requests going to parameter URLs; the Page indexing report shows new products sitting as “Discovered – currently not indexed” for weeks.

The fix isn’t a single setting. It is a sequence: stop linking filter combinations that nobody searches for, block those parameter patterns from crawling, make sure category pages link to every product, keep the product sitemap current, and check host status for slow responses during peak hours. Then watch Crawl Stats by purpose: discovery requests for new product URLs should rise as wasted requests fall, and the “Discovered – currently not indexed” count for the product sitemap should start to shrink.

This is general, not a client’s data, but it is the pattern I see most often. For the template side of the same problem, see enterprise technical SEO; for the audit method, how to audit a very large site.

A large site tree with most branches greyed out and blocked, and the remaining important branches highlighted in teal with a crawler path running through them
Figure 3. After the fixes: wasteful branches are closed, and crawling follows the paths to the pages that matter.

Free tool · SEO ROI Calculator

Estimate what the keywords behind your uncrawled page types are worth, so crawl fixes get the priority they deserve.

Enter search volume, CPC and your current ranking to estimate traffic, revenue and value for a keyword. Free.

Calculate your SEO ROI →

FAQ

What is crawl budget in SEO?

The set of URLs Google can and wants to crawl on your site. Google says it is determined by crawl capacity limit (how much your server can handle) and crawl demand (how much Google wants to crawl your URLs).

Does crawl budget matter for small websites?

Rarely. Google’s guide is aimed at large sites, very fast-changing sites, or sites with many URLs stuck as “Discovered – currently not indexed”. For smaller sites, keep the sitemap up to date and check the Page indexing report.

How do I check my crawl budget?

Use the Crawl Stats report in Search Console, grouped by response, file type, purpose and Googlebot type, plus server logs if you have them. Compare it with the Page indexing report for each sitemap.

What is SEO for large sites mostly about?

Templates, URL rules and crawling: making sure the pages that matter are linked, canonical, fast and in sitemaps, and that useless URL spaces don’t consume Google’s crawling.

Eugen Platon

Eugen Platon

Director of SEO & Web Analytics at onwardSEO
Eugen Platon is a highly experienced SEO expert with 16+ years of experience propelling organizations to the summit of digital popularity. Eugen, who holds a Master's Certification in SEO and is well-known as a digital marketing expert, has a track record of using analytical skills to maximize return on investment through smart SEO operations. His passion is not simply increasing visibility, but also creating meaningful interaction, leads, and conversions via organic search channels. Eugen's knowledge goes far beyond traditional limits, embracing a wide range of businesses where competition is severe and the stakes are great. He has shown remarkable talent in achieving top keyword ranks in the highly competitive industries of gambling, car insurance, and events, demonstrating his ability to traverse the complexities of SEO in markets where every click matters. In addition to his success in these areas, Eugen improved rankings and dominated organic search in competitive niches like "event hire" and "tool hire" industries in the UK market, confirming his status as an SEO expert. His strategic approach and innovative strategies have been successful in these many domains, demonstrating his versatility and adaptability. Eugen's path through the digital marketing landscape has been distinguished by an unwavering pursuit of excellence in some of the most competitive businesses, such as antivirus and internet protection, dating, travel, R&D credits, and stock images. His SEO expertise goes beyond merely obtaining top keyword rankings; it also includes building long-term growth and optimizing visibility in markets where being noticed is key. Eugen's extensive SEO knowledge and experience make him an ideal asset to any project, whether navigating the complexity of the event hiring sector, revolutionizing tool hire business methods, or managing campaigns in online gambling and car insurance. With Eugen in charge of your SEO strategy, expect to see dramatic growth and unprecedented digital success.
Eugen Platon
Check my Online CV page here: Eugen Platon SEO Expert - Online CV.