Crawl budget is the set of URLs a search engine can and wants to crawl on your site in a given period, set by how fast your server lets bots crawl and how much the engine actually wants your pages. For most small sites it is a non-issue. On large, templated or programmatic sites it decides whether your money pages get recrawled weekly or sit in “Discovered - currently not indexed” for months.
Crawl Budget
Crawl budget is the total number of URLs a search engine crawler will request from a site in a given period, set by the crawl rate limit (what the server can take) and crawl demand (what the engine wants), and shaped by site size, page quality, internal linking, and crawl directives.
What crawl budget actually is
Google defines crawl budget as “the set of URLs that Google can and wants to crawl”, built from two inputs. Neither is a dial you set; both are inferred from how your site behaves.
- Crawl capacity limit. The maximum fetch rate Googlebot will use without overloading your server. Fast, error-free responses raise it; slow responses, timeouts,
5xxerrors and429(rate limiting) lower it quickly. Google would rather under-crawl than take your site down. - Crawl demand. How much the engine wants your URLs, driven by popularity (links, traffic), perceived quality and staleness. A URL nobody links to and nobody updates earns very little demand, so it gets crawled rarely, sometimes once and never again.
Crawl budget is no ranking factor, and Google has said so. A higher crawl rate does not lift positions. What it governs is the throughput of discovery, indexing and refresh, and those decide which pages get the chance to rank.
The trap is treating crawl budget as something you hold in an account. It is an emergent property: you stop wasting it, and you earn more of it by being fast and worth crawling. For the mechanics, see how web crawlers work and our breakdown of Googlebot.
Who actually needs to care
Be honest about whether this applies to you before you spend a week on it. Google’s crawl budget guide is written for two kinds of site, which it calls a rough estimate: 1 million+ unique pages with content that changes about weekly, and 10,000+ unique pages whose content changes daily. Google also flags a third case at any size: a large share of your URLs sitting in “Discovered - currently not indexed”.
| Site profile | Crawl budget a priority? | Why |
|---|---|---|
| Brochure or local site (under 1,000 URLs) | No | Google crawls it in full, easily |
| Mid-size blog or SaaS (1,000 to 10,000 URLs) | Rarely | Only when duplication or a slow server inflates the crawl |
| Large e-commerce or faceted catalog | Yes | Filter and parameter URLs multiply the URL count |
| Programmatic site (10,000 to 1M+ URLs) | Critical | Discovery speed decides what gets indexed |
| News or fast-changing content | Yes | Freshness depends on recrawl speed |
If your pages get crawled the day they are published, you do not have a crawl budget problem, whatever an audit tool says. The conversation belongs to large, fast-changing or heavily templated sites, exactly the kind of programmatic SEO build where one bad URL pattern multiplies into a million crawlable dead ends.
What crawl budget gets spent on
Bots spend the budget on every URL they can reach, including the ones you do not care about. Every wasted fetch is a money page that waited. The usual culprits:
- Duplicate URLs: parameter variants (
?sort=,?ref=), session IDs, trailing-slash and case inconsistencies, HTTP and HTTPS or www and non-www splits. - Faceted navigation: filter combinations on catalogs that explode into combinatorial URL counts.
- Thin pages: tag archives, empty categories, autogenerated thin content.
- Redirect chains: every hop in a redirect chain is a separate fetch.
- Soft 404s and infinite spaces such as calendars and internal search results.
- Page resources: on JavaScript-heavy sites, the scripts, stylesheets and API calls Googlebot fetches to render a page also take crawl capacity.
When thousands of URLs sit in Discovered - currently not indexed, crawl waste is the first suspect: Google found them, decided they were not worth fetching yet, and parked them.
How to optimize your crawl budget
Order matters: fix waste and discovery before you think about crawl rate.
1. Cut the waste first
Most crawl budget problems are duplication problems in disguise.
- Consolidate duplicates with
rel="canonical"and pick one URL format (one casing, one trailing-slash rule, one protocol and host). - Block truly useless paths (internal APIs, tracking endpoints, infinite filter spaces) in
robots.txt. The free robots.txt generator writes the syntax so a typo does not block the wrong folder.robots.txtcontrols crawling only; to keep a page out of the index, let it be crawled and servenoindex(see the meta robots settings). Note thatnoindexsaves no crawl budget, because Google still requests the page before it drops it. - Prune or consolidate thin pages, and 301 the dead ones into stronger equivalents.
Do not use robots.txt to shuffle budget around temporarily. Google says freed capacity only moves to other pages when your site is already hitting its crawl capacity limit.
2. Control faceted navigation and parameters
On catalogs this is usually the whole game. Disallow low-value filter combinations, canonicalize the rest to the clean category URL, and stop linking to parameter URLs from your own navigation. A crawler cannot waste time on a URL it never finds.
3. Tighten architecture and internal linking
Crawlers follow links. Flat, well-linked sites get crawled efficiently; deep, orphaned ones do not.
- Keep important pages within a few clicks of the homepage (two or three is a good rule of thumb).
- Use contextual internal links to point crawlers at priority URLs. Our site structure guide goes deeper.
- Find and link orphaned content: a page with no internal links is close to invisible to crawlers.
4. Keep sitemaps honest
Your XML sitemap is a list of what you want crawled. List only canonical, indexable, 200-status URLs, strip out redirected and noindexed entries, and keep lastmod accurate so freshness signals mean something.
5. Make the server fast
The crawl capacity limit follows response time. Reduce time to first byte, cache aggressively, and put a CDN in front of static assets. Support 304 Not Modified responses, so Googlebot can reuse its cached copy of unchanged pages. Faster responses let Googlebot fetch more URLs in the same time, and they protect crawlability when traffic spikes.
6. Fix errors and redirect chains
Monitor 4xx and 5xx responses in Search Console, repair or 301 broken pages, and collapse multi-hop redirects into a single hop. A chain of three redirects costs three fetches to deliver one page.
7. Slow Googlebot only in an emergency
Google removed Search Console’s crawl rate limiter tool in January 2024, and Googlebot ignores the Crawl-delay line in robots.txt. If Googlebot is overloading your server, returning 500, 503 or 429 for a short time makes it back off on its own. Keep it short: weeks of errors cost you indexed pages. The lasting fix is a server that can take the crawl.
How to measure it
The signal lives in two places:
- Crawl Stats report (Search Console, Settings): total crawl requests, average response time, host status, and breakdowns by response code, file type and purpose. Rising response time or a growing
5xxshare means your crawl capacity is shrinking. - Server log files: the ground truth. Logs show exactly which URLs bots hit, how often, and which they ignore. This is where you catch crawls burned on parameter junk that the Search Console summaries hide.
For single URLs, see how to check when Google last crawled a page. Track indexation rate, response time and recrawl delay for priority URLs; total crawl counts on their own tell you little.
“Pages crawled went up 40%” means nothing if the extra crawls hit junk. The metric that matters is whether your money pages get discovered and refreshed faster.
When crawl budget becomes a business problem
Under a few thousand URLs, with new pages crawled within days, ignore crawl budget. Your time is better spent on content.
It becomes a business problem on large and programmatic sites: tens of thousands of templated pages, a “Discovered - currently not indexed” count that keeps growing, logs showing Googlebot busy on filters and parameters while new pages wait, or a launch of 20,000 pages that Google takes months to fetch. That is architecture work, and it is far cheaper to get right before the build than to clean up after. On our programmatic SEO projects we plan the URL set, the internal links and the sitemaps with the crawl in mind, then check the logs once pages ship. Gyfted’s templates per career, US state and industry reached 126,824 Google clicks in its best month.
Frequently Asked Questions
What is crawl budget in SEO?
Crawl budget is the set of URLs a search engine can and wants to crawl on your site in a given period. It is set by your crawl capacity limit (how fast your server lets bots crawl) and crawl demand (how much the engine wants your pages). It governs how quickly content gets discovered, indexed and refreshed.
Is crawl budget a ranking factor?
No. Google has said crawl rate is not a ranking factor. Crawl budget affects discovery and freshness: how fast new pages get indexed and how often existing ones get recrawled. Waste it on junk URLs and your important pages may be crawled rarely, which costs visibility indirectly.
Does my small website need to worry about crawl budget?
Almost certainly not. Google’s crawl budget guide is aimed at sites with 1 million+ pages that change weekly, or 10,000+ pages that change daily. If your new pages get crawled within a day or two, crawl budget is not your problem.
How do I check my crawl budget in Google Search Console?
Open Settings, then the Crawl Stats report. It shows total crawl requests over time, average server response time, and a breakdown by response code, file type, purpose and Googlebot type. Rising response times or a growing share of 5xx errors signal shrinking crawl capacity. Server logs give the full URL-level picture.
Does robots.txt save crawl budget?
Yes, for paths you never want crawled: a disallowed URL is not fetched. It does not remove URLs already in the index, and blocked URLs can still appear in search without a snippet. noindex saves nothing, because Google has to fetch the page to read it. Use robots.txt for waste and noindex for index control.
How do I increase my crawl budget?
Raise both inputs. Capacity grows with fast, error-free responses. Demand grows with links, real popularity and content that changes in ways worth recrawling. Cutting duplicate and thin URLs frees the existing budget for pages that matter, which is usually the quickest gain.