Web crawlers — also called spiders or bots — are automated programs that move through the web by following links, fetching pages, and adding new URLs to a queue so search engines can later index and rank them. They start from a list of known URLs, request each page, read the HTML and links inside it, and repeat that loop across billions of pages. If a crawler never reaches your page, nothing downstream matters: no indexing, no ranking, no traffic.
How Web Crawlers Work
A web crawler is an automated bot that systematically discovers, fetches, and queues web pages by following hyperlinks and sitemap entries, supplying the raw content that search engines use to build their index.
The crawl cycle: discover, fetch, parse, queue
Every crawler — Googlebot, Bingbot, or a third-party SEO tool — runs the same core loop. Understanding it tells you exactly where pages get lost.
1. Discover. The crawler builds a list of candidate URLs from seeds it already knows: previously crawled pages, XML sitemaps you submit, internal links, and backlinks pointing at your site. New URLs enter the system almost entirely through links and sitemaps — which is why an orphaned page with zero inbound links is effectively invisible.
2. Fetch. The bot sends an HTTP(S) request with a user-agent string (so you can identify it in your logs) and downloads the response. It handles status codes along the way: 200 means serve and parse, 301/302 means follow the redirect, 404/410 means drop the URL, and 5xx triggers retry-with-backoff. Long redirect chains waste crawl resources and dilute the signal that reaches the destination.
3. Parse and render. The crawler reads the HTML to extract links, the title, meta description, canonical tag, structured data, hreflang, and robots meta directives. When content depends on client-side JavaScript, the page goes into a render queue where a headless browser executes the script to surface links and text that aren’t in the raw HTML. That render step is slower and rationed — which is the whole reason JavaScript-heavy sites get crawled less completely.
4. Queue and prioritize. Newly discovered URLs go into the crawl frontier — a prioritized queue. The crawler decides what to fetch next based on importance signals (internal links, backlinks, depth from the homepage) and freshness signals (Last-Modified headers, sitemap lastmod, observed change frequency). Then the loop repeats.
The crawl loop is mechanical and relentless, but it is not infinite. Every site gets a finite share of attention. The job of technical SEO is to spend that share on pages that matter.
What crawlers obey: the rules of the road
Crawlers aren’t free to grab everything. Well-behaved bots respect a stack of directives, and getting these wrong is one of the most common ways teams accidentally block themselves.
| Control | Where it lives | What it does |
|---|---|---|
robots.txt | /robots.txt at the root | Allows or disallows crawling of paths. Blocks the fetch, not indexing. |
| Meta robots | <head> of the page | noindex, nofollow — controls indexing and link-following after fetch. See meta robots settings. |
| X-Robots-Tag | HTTP response header | Same directives as meta robots, but works for PDFs, images, and non-HTML files. |
Crawl-delay / rate limits | Server + crawler logic | Throttles request frequency to avoid overloading the origin. |
| Canonical tag | <head> link element | Tells the crawler which URL is the master version among duplicates. |
The single most important nuance: robots.txt blocks crawling, not indexing. A URL disallowed in robots.txt can still appear in search results (URL-only, no snippet) if other pages link to it. To actually keep a page out of the index, you need a noindex directive the crawler can read — which means the page must be crawlable. Blocking it in robots.txt prevents the bot from ever seeing the noindex. That contradiction is behind a huge share of “why is this indexed?” tickets.
Politeness, crawl budget, and traps
Crawlers throttle themselves so they don’t behave like a denial-of-service attack. They cap concurrent connections per host, space out requests, and back off when your server returns errors or slows down. The flip side: if your server is fast and healthy, Google will crawl more; if it’s flaky, Google crawls less. Server performance is a crawl signal.
That self-imposed limit is what we call crawl budget — the number of URLs a search engine is willing to fetch from your site in a given window. It rarely constrains small sites. It absolutely constrains large ones, where crawlers can drown in:
- Crawler traps — infinite calendars, endless faceted-navigation combinations, and session-ID URLs that generate near-limitless unique addresses.
- Parameterized URL sprawl —
?sort=,?ref=,?utm=variants that multiply one page into thousands. - Soft 404s and thin pages that consume fetches without earning a place in the index.
The fix is structural: clean site architecture, URL normalization, canonical tags to collapse duplicates, and disallowing genuinely worthless paths so the bot spends its budget on pages you want ranked.
Crawled is not indexed is not ranked
A frequent and expensive misconception is treating “crawled” as the finish line. It’s three distinct stages:
- Crawled — the bot fetched the URL.
- Indexed — the engine stored a processed version of it and deemed it worth keeping. Plenty of crawled pages never make it; the Discovered – currently not indexed status in Search Console is exactly this gap.
- Ranked — the page competes for positions on a query.
This page covers stage one: discovery and crawling. Once a page is in the index, an entirely separate system decides where it ranks. For that side of the pipeline — relevance, authority, E-E-A-T, and the signals behind position — see search engine ranking. For how Google’s specific crawler behaves, see what is Googlebot, and for the upstream prerequisite of being reachable at all, crawlability.
How we make sites crawl-efficient
When we run a technical pass, crawl efficiency is the first thing we check — because no amount of content or links helps a page the bot can’t reach. The high-leverage moves, in order:
- Submit and maintain an XML sitemap with accurate
lastmodvalues so crawlers find new and updated URLs fast. - Flatten internal linking so important pages sit a few clicks from the homepage, never buried or orphaned.
- Audit
robots.txtandnoindexfor self-inflicted blocks — the classic is shipping a staging-environmentDisallow: /to production. - Kill duplicate and parameter sprawl with canonicals and consistent URL formatting.
- Keep the server fast and stable, because response time directly shapes how aggressively Google crawls. You can confirm when a URL was last fetched by checking the last-crawl date.
In the AI Overviews era, this matters even more: the same crawl-and-index pipeline that feeds the classic SERP also feeds the generative answers and the AI crawlers pulling content for LLM training and retrieval. If your pages aren’t crawlable, they’re absent from both. This is the foundation our AI SEO services and programmatic SEO program are built on — efficient crawling at scale before anything else.
Frequently Asked Questions
What is the difference between crawling and indexing?
Crawling is when a bot fetches a page by following a link or sitemap entry. Indexing is when the search engine processes that page and stores it in its database as eligible to appear in results. Crawling happens first; many crawled pages are never indexed because the engine judges them low-value or duplicate.
How do I get Google to crawl my site faster?
Submit an XML sitemap with accurate lastmod dates, build strong internal links so important pages aren’t buried, earn backlinks from already-crawled sites, and keep your server fast and error-free. A healthy, responsive site signals Google to crawl more often. For a single URL, use the URL Inspection tool in Search Console to request indexing.
Does robots.txt stop a page from being indexed?
No. The robots.txt file blocks crawling, not indexing. A blocked URL can still appear in results if other pages link to it, because Google sees the link but can’t read the page’s content. To keep a page out of the index, allow it to be crawled and add a noindex meta robots or X-Robots-Tag directive the crawler can actually read.
What is crawl budget and does it affect my site?
Crawl budget is the number of URLs a search engine will fetch from your site in a given period. It rarely limits small sites — a few thousand pages get crawled easily. It matters for large sites where crawler traps, parameter sprawl, and thin pages waste the budget on URLs that don’t deserve indexing, starving the pages that do.
Can search engines crawl JavaScript content?
Yes, but with a delay and at a cost. Crawlers fetch the raw HTML first, then queue pages needing JavaScript for a separate rendering step using a headless browser. Because rendering is slower and resource-limited, JavaScript-dependent content is crawled less completely. Server-side rendering or pre-rendering critical content keeps it reliably visible to bots.