I’m not an SEO guy, and the thing that trips up most people who are SEO people is that crawling and indexing are two different problems with two different toolkits. You can block crawling without affecting indexing. You can have a page fully indexed that you never wanted indexed. You can have pages Google finds, crawls, and then decides not to index — and the reason lives in a dropdown in Search Console that most people don’t know to check. Getting control of the crawl and indexing layer is the mechanical foundation on which everything else in SEO sits. Get it wrong and you’re optimising content that Google isn’t reading, or earning links to pages Google has decided aren’t worth keeping.
This post follows the trail that Part 1 (“How Google Search Actually Works”) opened — the crawl stage in detail — and feeds into Part 5 (“Technical SEO and Structured Data”), where indexing quality determines whether your structured data ever reaches a rich result. The source for all of this is Google’s own crawling-indexing documentation (last updated December 2025 across the section index), plus Google’s current guidance on robots.txt, sitemaps, canonicalization, and crawl budget. I’m not going to invent statistics. Every claim traces to a source.
How Googlebot Decides What to Crawl and When
Googlebot discovers URLs through a handful of mechanisms: sitemaps you submit, links on pages it already knows about, and direct submission via URL Inspection in Search Console. Discovery is not the bottleneck for most sites. The bottleneck is what Googlebot decides to do after it knows a URL exists.
Two terms get conflated constantly: crawl rate and crawl budget. They’re not the same thing.
Crawl rate is how fast Googlebot crawls — how many simultaneous requests it makes, how long it waits between them. Google automatically adjusts this to avoid overloading your server. You can request a lower crawl rate in Search Console if you’re seeing server stress from Googlebot, but you cannot request a higher one; Google decides when your server can handle more. For the vast majority of sites this number is irrelevant because crawl demand doesn’t come close to the site’s actual capacity.
Crawl budget is the set of URLs Google considers worth crawling — and recrawling — over a given time window. This is the constraint that matters on large sites. Google doesn’t crawl an infinite number of pages. It makes prioritisation decisions based on signals including page quality, how recently a page changed, how many links point to it, and how many duplicate or near-duplicate versions exist in your URL space. The more URL noise you generate — parameter combinations, faceted navigation, session IDs, printer-friendly versions — the more crawl budget you dilute.
For a site with a few hundred pages, crawl budget almost certainly doesn’t constrain your visibility. For a large e-commerce site with tens of thousands of SKUs, a faceted navigation generating millions of URL combinations, or a programmatic SEO build pushing out thousands of landing pages at once, it is the central technical problem. Google’s documentation explicitly names faceted navigation as a known crawl budget sink.
The diagnostic entry point is Google Search Console’s “Crawl Stats” report. It shows you how often Google crawls your site, what response codes it’s hitting, and how much of the crawl is going to pages you actually care about versus URL bloat. If Googlebot is consuming a significant chunk of its crawl allocation on 404s, redirect chains, or parameter URLs you don’t need indexed, that’s the problem to fix first.
robots.txt: What It Does (and the Misconfiguration That Buries Sites)
robots.txt is a plain text file at yourdomain.com/robots.txt that tells crawlers which URLs they should and shouldn’t request. Google respects it, with qualifications.
Here is the distinction that saves sites from their own robots.txt:
Blocking crawl is not the same as blocking indexing.
A page blocked in robots.txt won’t be crawled. But Google can still index it — meaning it can appear in search results — if other pages link to it. Google knows the URL exists from those links; it just can’t read the content. The result is a search result listing that shows the URL with no title, no description, just “A description for this result is not available because of this site’s robots.txt” — which is embarrassing and often more harmful than just leaving the page accessible.
If you want to prevent a page from appearing in search results at all, you need a noindex tag on the page itself. robots.txt blocks the crawl; noindex blocks the indexing. They solve different problems and you frequently need both, independently.
The second common robots.txt error is blocking CSS and JavaScript files. Googlebot renders pages using a version of Chromium and needs to fetch and execute your JS and CSS to see what a user would see. Block those files and Google’s rendering pipeline gets an incomplete picture — which means it may assess your content differently from how it actually presents to users. Google’s guidance is explicit: don’t block CSS and JavaScript in robots.txt.
A robots.txt entry looks like this:
User-agent: Googlebot
Disallow: /admin/
Disallow: /search?The Disallow: /search? line is useful: it blocks crawling of any URL that starts with /search?, which is a clean way to handle parameter-driven search result pages you don’t want Googlebot spending time on. Test your robots.txt using Google Search Console’s robots.txt Tester before deploying changes. It will tell you exactly which URLs a given rule blocks, which prevents the “I thought I was blocking X but I was actually blocking Y” situation that has tanked more than a few site migrations.
Sitemaps: The Honest Guide to What They Do and Don’t Do
An XML sitemap is a file that lists URLs you want Google to know about. It is a crawl hint, not a crawl guarantee. Submitting a URL in your sitemap does not mean Google will index it, and it does not mean Google will crawl it on any schedule you’d consider timely.
What sitemaps do accomplish: they make URL discovery faster and more reliable, especially for deep or newly-published pages that haven’t yet accumulated many internal links. For a content site publishing frequently, or for a programmatic SEO build, sitemaps are the primary mechanism for getting new URLs into Google’s awareness quickly. Without a sitemap, Googlebot has to find those pages by following links — which is slower, especially if your internal linking structure is shallow.
What to include in your sitemap: URLs you want indexed. That’s the complete answer. Don’t include URLs that are blocked in robots.txt (it’s a contradiction Google will flag). Don’t include pages you’ve tagged with noindex. Don’t include redirect sources — include the final destination URLs. Don’t include duplicates.
Sitemap index files let you split large sitemaps into multiple files, with the index pointing to each. Google’s current limit is 50,000 URLs per sitemap file and 50MB uncompressed. Most sites never approach this. Large e-commerce sites and programmatic SEO builds hit it regularly, and the sitemap index structure is the correct solution — one index file listing separate sitemaps by content type (pages, products, posts, etc.).
Submit your sitemap in Google Search Console under “Sitemaps.” The console shows you the submitted count versus the indexed count. That gap — URLs you’ve submitted that haven’t been indexed — is useful diagnostic information. It doesn’t mean something is broken, but it’s the number to monitor over time. If it’s growing (more submitted, same or fewer indexed), something in your content quality or duplication situation is telling Google these pages aren’t worth keeping.
Keep your sitemap accurate. Remove URLs when you delete or noindex pages. A sitemap full of 404s or redirects is noise, and while Google is tolerant of it, there’s no reason to make Googlebot process URLs it can’t usefully crawl.
Canonical Tags: Resolving Duplicate Content Without Panic
Duplicate content is a structural feature of almost every non-trivial website, not a pathological condition. Any site with pagination, sort-and-filter URLs, print versions, HTTPS and HTTP coexistence, trailing slash variations, or URL parameters generates multiple URLs that resolve to the same or very similar content. The canonical tag is how you tell Google which version is the one that matters.
<link rel="canonical" href="https://www.example.com/page/" />That tag, in the <head> of the page, tells Google: “If you’re deciding which URL to index and show in search results, use this one.” Google describes canonicalization as selecting the “canonical” URL from a set of duplicates to represent the group.
A few things to know about how this actually works in practice:
Google treats rel=canonical as a strong hint, not a directive. If you point a canonical at a URL that’s blocked in robots.txt, or at a URL that itself has a different canonical, or at a URL that redirects away, Google may ignore your declared canonical and pick its own. The process of Google choosing differently from what you declared is what generates the “Duplicate without user-selected canonical” and “Duplicate, Google chose different canonical than user” statuses in Search Console — both of which are worth investigating when you find them.
Canonicalization status codes and the “Discovered — currently not indexed” status in Search Console are related but not the same problem. “Discovered” means Google knows the URL but hasn’t crawled it yet or hasn’t decided to index it. “Indexed, not submitted in sitemap” means Google found and indexed a URL you didn’t explicitly list. “Duplicate without user-selected canonical” means Google found multiple versions and you haven’t told it which one to prefer. Each of these is a distinct signal requiring a distinct response.
The short practical rule: every page that should appear in search results should have a self-referencing canonical tag pointing to the clean, preferred version of its URL. Every page that shouldn’t appear in search results should have noindex, not a canonical pointing elsewhere — because canonicalisation is for deduplication, not for removal.
On the rel=canonical versus 301 redirect question: if a URL is permanently gone and you want its link equity consolidated on the destination, 301 is the right tool — it’s a server-level signal, not a page-level hint, and Google treats it as authoritative. Canonical is for when both URLs should remain accessible (e.g., a filter URL you want to keep working but not index). Don’t use canonical to replace a 301 you’re just not bothered setting up.
Crawl Budget: When It Actually Matters
For a site with a few hundred pages and no URL bloat, crawl budget is not your problem. Google will crawl your whole site regularly and that’s the end of it.
The situations where crawl budget becomes a real constraint:
Large e-commerce sites with faceted navigation are the textbook case. A site with 10,000 products and faceted filtering by color, size, brand, and price can generate millions of unique URLs. Googlebot may crawl a small fraction of them. The answer is to either block the parameter URLs in robots.txt (crawl block) or noindex + canonical them to the clean category URL (index block), depending on whether you ever want a filter page to rank. The worst outcome is letting Googlebot crawl thousands of thin, nearly-identical filter pages and leaving it to Google to sort out the duplication — it will, eventually, but it’s expensive crawl budget spent on pages that will never rank.
Programmatic SEO builds have the same structural problem. If you’re publishing thousands of landing pages, the quality signal on those pages is what determines how much crawl budget Google allocates. A thousand high-quality, distinct pages will get indexed. A thousand thin, templated pages where the content barely differs across entries will generate “Discovered — currently not indexed” at scale, because Google crawls a sample, decides the quality isn’t there, and deprioritises the rest.
Redirect chains and 404s are quiet budget sinks. Every time Googlebot follows a redirect chain — URL A → URL B → URL C — it expends crawl budget on the hops. Long chains slow down the crawl and sometimes cause Googlebot to stop following before reaching the destination. Audit and flatten redirect chains as part of any site migration. And every time Googlebot crawls a 404, that’s budget spent on nothing. Clean up internal links pointing to deleted pages.
The key control levers for crawl budget management are: robots.txt to exclude URL patterns you never want crawled (parameter URLs, internal search, admin paths), canonical tags and noindex for pages that should be accessible but not indexed, accurate sitemaps that list only indexable URLs, and clean internal link structures that don’t lead Googlebot to dead ends or redirect chains.
Mobile-First Indexing: The Part That’s No Longer Optional
Mobile-first indexing is now the default for all sites. This is no longer a phased rollout or a “getting ready” situation — as of Google’s current documentation (December 2025), Googlebot primarily uses the mobile version of your site for indexing and ranking.
What this means in practice:
If your mobile version has less content than your desktop version — abbreviated text, missing structured data, lazy-loaded images that Googlebot can’t render — the mobile version is what gets indexed. The desktop version is largely irrelevant to Google’s evaluation. This is a hard reversal of the assumption that drove SEO practice for years.
The specific requirements Google documents: your mobile and desktop versions should have the same primary content, headings, structured data, and links. If you have a separate m. subdomain rather than a responsive design, you need to verify that Googlebot-Smartphone can access it and that both versions are linked with the appropriate alternate/canonical relationship. If your mobile version hides content behind “Read more” toggles or JavaScript interactions that Googlebot can’t expand, that content may not be indexed.
Server response time feeds directly into crawlability — if you run WordPress on a tuned LEMP stack (see the self-hosting WordPress on Hetzner guide), the FastCGI cache is what keeps Googlebot-Smartphone requests fast and properly rendered. A slow origin gets crawled less. The infrastructure layer and the indexing layer are not separate concerns.
Test your mobile rendering using the URL Inspection tool in Google Search Console — specifically the “Test Live URL” option, which uses Googlebot’s actual rendering. It will show you exactly what Googlebot sees, not what your browser shows after your network has delivered every asset. The gap between those two views is where mobile-first indexing problems live.
Frequently Asked Questions
What is the difference between robots.txt and noindex?
robots.txt blocks Googlebot from crawling a URL — requesting it. A noindex tag blocks Google from including a page in its index — search results. Blocking crawl doesn’t prevent indexing if other pages link to the URL. To reliably remove a page from search results, you need noindex on the page itself, not a robots.txt block.
How do I fix “Discovered — currently not indexed” in Search Console?
It means Google knows the URL exists but hasn’t prioritised crawling or indexing it. Common causes: low perceived quality relative to similar pages, crawl budget constraints, or thin content. Fix: improve content distinctiveness, check for canonicalization conflicts, confirm the URL is in your sitemap and not robots.txt-blocked. Site-wide, the signal is usually content quality or URL bloat.
Does my small site need to worry about crawl budget?
Almost certainly not. Crawl budget constraints are meaningful at scale — large e-commerce with faceted navigation, programmatic SEO at thousands of pages, or sites generating URL parameter combinations. For a site with a few hundred pages and clean URL structure, Google will crawl everything it finds without constraints. Focus on content and technical fundamentals instead.
How do I submit a sitemap to Google Search Console?
Go to Search Console → your property → Sitemaps (left nav) → enter the sitemap URL (usually /sitemap.xml or /sitemap_index.xml) → Submit. The console will show the count of URLs submitted and the count indexed. Discrepancies between those numbers are worth investigating over time — not immediately alarming, but a trend worth watching.
What happens if my canonical tag and redirect point to different URLs?
Google follows the redirect first; the redirect takes precedence as a server-level signal. If A canonicals to B but A redirects to C, Google will typically treat C as the canonical — which may not be what you intended. Audit your canonical and redirect pairs after any site restructure. Contradictions between the two generate confusion in Search Console and slow down canonical resolution across your site.
Crawl and Indexing Problems Are Easy to Get Wrong
The mechanics here aren’t complicated. The errors are. A robots.txt block on a CSS file silently breaks rendering without throwing a visible error. A noindex tag on a canonical destination removes the page you intended to rank. A redirect chain three hops long quietly eats crawl budget for months before anyone checks the Crawl Stats report. A sitemap full of noindexed pages tells Google you don’t understand your own site.
Crawl and indexing control compounds over time. Every week Googlebot spends on the wrong pages is budget not spent on the pages that matter. Every canonicalisation conflict is a ranking signal diluted across multiple URLs. These problems don’t announce themselves — you find them in Search Console data, in crawl logs, in the gap between submitted sitemap URLs and indexed counts.
If your site is large, recently migrated, or generating coverage issues you can’t attribute — our Fractional SEO service starts with a crawl and indexing audit as the foundation. We map the gap between what Googlebot is finding and what you actually want it to find, then fix the mechanical layer before touching anything else. The right foundation makes every subsequent SEO investment more effective.