Web crawlers (also called bots, spiders or robots) are automated programs that move through the web by following links, fetching pages, and adding the new URLs they find to a queue so search engines can later index and rank them. They start from a list of known URLs, request each page, read the HTML and the links inside it, and repeat that loop across billions of pages. If a crawler never reaches your page, nothing downstream happens: no indexing, no ranking, no traffic, and no citation in an AI answer.
How Web Crawlers Work
A web crawler is an automated bot that systematically discovers, fetches, and queues web pages by following hyperlinks and sitemap entries, supplying the raw content that search engines use to build their index.
The crawl cycle: discover, fetch, parse, queue
Every crawler (Googlebot, Bingbot, or a third-party SEO tool) runs the same core loop. Understanding it tells you exactly where pages get lost.
1. Discover. The crawler builds a list of candidate URLs from what it already knows: previously crawled pages, XML sitemaps you submit, internal links, and backlinks pointing at your site. New URLs enter the system almost entirely through links and sitemaps, which is why an orphaned page with no inbound links is close to invisible.
2. Fetch. The bot sends an HTTP request with a user-agent string (so you can identify it in your logs) and downloads the response. Status codes steer what happens next: 200 means parse the page, 301/302 means follow the redirect (Googlebot follows up to 10 hops), 404/410 means the URL drops out of the index and gets recrawled less and less, and 5xx or 429 makes the crawler slow down and retry later. Long redirect chains cost a fetch per hop.
3. Parse and render. The crawler reads the HTML to extract links, the title, meta description, canonical tag, structured data, hreflang and robots directives. When content depends on client-side JavaScript, Google puts the page in a rendering queue, where a headless Chromium runs the scripts to surface text and links missing from the raw HTML. Rendering is a separate, later step, so JavaScript-heavy sites depend on it working and on it happening at all.
4. Queue and prioritize. Newly discovered URLs go into the crawl frontier, a prioritized queue. The crawler decides what to fetch next from importance signals (internal links, backlinks, depth from the homepage) and freshness signals (sitemap lastmod, Last-Modified headers, how often the page has changed before). Then the loop repeats.
The crawl loop is mechanical and relentless, and it is finite. Every site gets a limited share of attention. The job of technical SEO is to spend that share on pages that matter.
What crawlers obey: the rules of the road
Well-behaved bots respect a stack of directives, and getting these wrong is one of the most common ways teams block themselves by accident.
| Control | Where it lives | What it does |
|---|---|---|
robots.txt | /robots.txt at the root | Allows or disallows crawling of paths. It controls the fetch only, so a blocked URL can still be indexed. |
| Meta robots | <head> of the page | noindex, nofollow: controls indexing and link-following after the fetch. See meta robots settings. |
| X-Robots-Tag | HTTP response header | Same directives as meta robots, for PDFs, images and other non-HTML files. |
Crawl-delay | robots.txt | Asks for a pause between requests. Bing and Yandex honor it; Googlebot ignores it and sets its own rate from how your server responds. |
| Canonical tag | <head> link element | A hint naming the preferred URL among duplicates. |
The nuance that causes the most trouble: robots.txt controls crawling only. A URL disallowed in robots.txt can still appear in search results (URL only, no snippet) if other pages link to it. To keep a page out of the index, it needs a noindex the crawler can read, which means the page must stay crawlable. Blocking it in robots.txt stops the bot from ever seeing the noindex. That contradiction is behind a large share of “why is this indexed?” tickets.
User agents can be faked, so before you trust a log line that says Googlebot, verify it with a reverse DNS lookup that resolves to googlebot.com or google.com, or against Google’s published IP ranges.
Politeness, crawl budget, and traps
Crawlers throttle themselves so they do not behave like a denial-of-service attack. They cap connections per host, space out requests, and back off when your server returns errors or slows down. The flip side: a fast, healthy server lets Google crawl more when it wants to, and a flaky one gets crawled less. Server performance is a crawl signal.
That limit, combined with how much the engine wants your URLs, is your crawl budget. It rarely constrains small sites. On large ones, crawlers can drown in:
- Crawler traps: infinite calendars, endless faceted-navigation combinations, and session-ID URLs that generate near-limitless unique addresses.
- Parameter sprawl:
?sort=,?ref=,?utm=variants that multiply one page into thousands. - Soft 404s and thin pages that consume fetches without earning a place in the index.
The fix is structural: clean site architecture, URL normalization, canonical tags to collapse duplicates, and disallowing truly worthless paths so the bot spends its time on pages you want ranked.
AI crawlers: same loop, different bots
ChatGPT, Claude and Perplexity run their own crawlers, with their own user agents: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot. They follow the same discover and fetch loop and read your robots.txt under their own names, so a rule written for Googlebot says nothing about them. Google’s AI Overviews and AI Mode use Googlebot’s crawl, and Google-Extended is a robots.txt token with no crawler of its own that controls Gemini training and grounding.
Two practical differences. First, in a December 2024 analysis of its network, Vercel found that the major AI crawlers fetched JavaScript files without executing them, so content that exists only after client-side rendering was invisible to them; server-side rendering covers both. Second, CDN and firewall bot rules often block these crawlers without anyone deciding to. Paste your robots.txt into the free AI-bot access checker to see, bot by bot, which ones your rules let in.
Crawled, indexed and ranked are three different stages
A frequent and expensive mistake is treating “crawled” as the finish line.
- Crawled: the bot fetched the URL.
- Indexed: the engine stored a processed version and judged it worth keeping. Plenty of crawled pages never make it; Discovered - currently not indexed and “Crawled - currently not indexed” in Search Console are exactly this gap.
- Ranked: the page competes for positions on a query.
This page covers stage one. Once a page is in the index, a separate system decides where it ranks; for that side, see search engine ranking. For Google’s own crawler, see what is Googlebot, for the prerequisite of being reachable at all, crawlability, and for a longer walk through the whole pipeline, how Google Search works.
How to make a site easy to crawl
When we run a technical pass, crawl efficiency comes first, because no amount of content or links helps a page the bot cannot reach. The high-leverage moves, in order:
- Submit and maintain an XML sitemap with accurate
lastmodvalues so crawlers find new and updated URLs fast. - Flatten internal linking so important pages sit a few clicks from the homepage and none are orphaned.
- Audit
robots.txtandnoindexfor self-inflicted blocks; the classic is shipping a stagingDisallow: /to production. - Collapse duplicate and parameter URLs with canonicals and consistent URL formatting.
- Keep the server fast and stable, because response time shapes how hard Google crawls. You can confirm when a URL was last fetched by checking the last-crawl date.
- Put critical content and links in the server-sent HTML, so every crawler, including the ones that do not render JavaScript, sees them.
When crawling becomes a business problem
On a small site with clean HTML and a sane robots.txt, crawlers find everything on their own. A quick look at Search Console’s Page indexing report once a month is enough, and you do not need us.
It costs money when the bots are missing pages that earn revenue. The symptoms: new product or landing pages that take weeks to show up, a growing pile in “Discovered - currently not indexed”, content that only exists after JavaScript runs, a CDN rule quietly blocking Bingbot or the AI crawlers, or Bing and Google telling two different stories about the same site. On a fractional SEO engagement the two founders find which crawler misses what, in your logs and both engines’ reports, and fix the cause. On Prawomat’s launch, Bing ranked the young domain months before Google trusted it, which is why we read both engines.
Frequently Asked Questions
What are web crawlers also called?
Bots is the common short name. Crawlers are also called spiders, robots or web crawlers, and each search engine names its own: Googlebot for Google, Bingbot for Bing. AI companies run their own, such as GPTBot and ClaudeBot.
What is the difference between crawling and indexing?
Crawling is when a bot fetches a page by following a link or sitemap entry. Indexing is when the search engine processes that page and stores it as eligible to appear in results. Crawling comes first, and many crawled pages are never indexed because the engine judges them low-value or duplicate.
How do I get Google to crawl my site faster?
Submit an XML sitemap with accurate lastmod dates, build strong internal links so important pages are close to the homepage, earn links from sites Google already crawls often, and keep your server fast and error-free. For a single URL, use URL Inspection in Search Console to request indexing.
Does robots.txt stop a page from being indexed?
No. The robots.txt file controls crawling only. A blocked URL can still appear in results if other pages link to it, because Google sees the link and cannot read the page. To keep a page out of the index, let it be crawled and add a noindex meta robots tag or X-Robots-Tag header.
What is crawl budget and does it affect my site?
Crawl budget is the set of URLs a search engine can and wants to crawl on your site. It rarely limits small sites. It matters on large sites, where crawler traps, parameter sprawl and thin pages use up fetches that the pages you care about need.
Can search engines crawl JavaScript content?
Google can: it fetches the raw HTML first, then renders pages that need JavaScript in a later step with a headless Chromium. Many other crawlers, including the major AI crawlers in Vercel’s 2024 analysis, do not execute JavaScript at all. Server-side rendering or pre-rendering critical content keeps it visible to every bot.