Googlebot is the automated program Google uses to discover, fetch, and process web pages so they can be considered for Google Search. It works in three phases (crawling, rendering, indexing) and obeys signals like robots.txt, meta robots, and canonical tags along the way. If a page is broken, blocked, or unrenderable at any of those phases, it never makes the index, which is why understanding Googlebot is foundational to every technical SEO call we make.
Googlebot
Googlebot is Google’s web crawler: a family of user-agents that fetch pages, follow links, and render content, then hand it to Google’s indexing systems for inclusion and ranking in Search.
How Googlebot actually works
Googlebot works as a pipeline of three distinct stages, each with its own failure modes. Treating it that way is the difference between “we requested indexing and prayed” and actually diagnosing why a URL won’t rank.
Stage 1: Crawling. Googlebot maintains a queue of known URLs from past crawls, your XML sitemaps, and links found across the web. It requests each URL over HTTP(S), reads the response code, and queues new links. It’s the same mechanic behind how web crawlers work generally, at planetary scale, with Google’s priorities.
Stage 2: Rendering. Modern Googlebot runs an evergreen, headless Chromium build. After the raw HTML fetch, render-needing pages enter a queue where JavaScript executes and the DOM is built out. The rendered DOM, which goes beyond the source HTML, is what gets indexed. Content that only appears after a client-side fetch counts only once rendering completes.
Stage 3: Indexing. Google parses the rendered page: content, structured data, title and meta tags, canonical signals, hreflang, and links. It decides whether to index, which URL is canonical, and how to represent the page. A noindex seen here pulls the page out, even if it crawled fine. If the noindex is already in the raw HTML, Google may skip rendering that page, so JavaScript that removes the tag later may never run.
The trap we see constantly: people optimize for crawling and forget rendering. A page returns a clean 200, shows up in your logs, and still won’t index, because the content Googlebot needs is gated behind JavaScript that never runs.
The Googlebot user-agents
“Googlebot” is shorthand for several distinct crawlers. The ones that matter for organic SEO:
| User-agent | Primary job | Notes |
|---|---|---|
| Googlebot Smartphone | Crawls/indexes the mobile view | The default since mobile-first indexing rolled out site-wide; this is what indexes your content |
| Googlebot Desktop | Crawls the desktop view | Now secondary; runs far less often than the smartphone agent |
| Googlebot Image / Video | Media discovery | Feeds Google Images and video surfaces |
| Google-InspectionTool | Powers URL Inspection & Rich Results Test | Has its own user-agent but follows your Googlebot rules in robots.txt; it runs when you test a URL |
| Google-Extended | AI training/grounding control | A robots.txt token with no crawler of its own. Controls Gemini training and grounding use of content Google crawls; no effect on Search |
Two practitioner-grade points. First: because indexing is mobile-first, if content, structured data, or links exist on desktop but not mobile, Google effectively doesn’t see them. Second: Google-Extended is a robots.txt token with no crawler of its own. AI features in Google Search, including AI Overviews, follow your Googlebot rules; Google-Extended controls Gemini training and grounding use and does not affect Search. Block it if you want your content kept out of Gemini training and grounding; it saves no crawling.
What controls Googlebot: the signals that matter
Googlebot reads directives in a specific order. Get the order wrong and you’ll block the very signal you needed Google to see.
- robots.txt: Checked before fetching. A
Disallowstops the crawl. Critical gotcha: a blocked URL can still get indexed (URL-only, no snippet) from external links, because Google never fetched the page to see yournoindex. Use robots.txt to manage crawl; deindexing needs a different tool. To write the file without syntax mistakes, our free robots.txt generator builds the groups and the sitemap line for you. - Meta robots & X-Robots-Tag: Read after fetching, so the page must be crawlable for these to work.
noindexhere is the correct deindexing tool. See our deep-dive on meta robots advanced settings and the X-Robots-Tag HTTP header for non-HTML files. - Canonical tags: A hint Google weighs, with no obligation to follow it. Google picks the canonical from a cluster of signals; your
rel=canonicalis one vote. Conflicting signals cause the duplicate without user-selected canonical status in Search Console. - HTTP status codes: 200 gets processed, 3xx redirects are followed (mind redirect chains), 404 and 410 drop the page over time, while 429 and repeated 5xx errors slow crawling and, if they persist, can deindex. Watch for soft 404s, which waste crawl on pages Google should never have queued.
Crawl budget: who actually needs to care
Crawl budget is the number of URLs Googlebot will fetch on your site in a given window. It’s governed by two things: crawl capacity (how much your server can handle without slowing down or erroring) and crawl demand (how much Google wants your URLs, driven by popularity and freshness).
Google says it plainly in its own crawl budget guide: the topic is for large sites, roughly a million or more unique pages that change weekly, or 10,000 or more that change daily. Most sites can ignore crawl budget. If a small site isn’t indexing, it’s almost always a quality, rendering, or directive issue. Budget becomes a real lever on large catalogs, faceted-navigation sites, and anything generating infinite parameter URLs.
| Wastes crawl budget | Protects crawl budget |
|---|---|
| Faceted/parameter URL explosion | Canonicalization + parameter discipline |
| Long redirect chains | Single-hop 301s |
| Soft 404s and thin content | Pruning or noindexing low-value pages |
| Orphaned content Google can’t reach | Strong internal links + clean sitemaps |
| Slow server responses / 5xx spikes | Fast, stable hosting |
The goal of budget work is to make Google spend its crawl on the URLs that earn money. That’s a site structure and pruning job, and no single robots.txt line will do it for you.
How to verify and monitor Googlebot
Anyone can spoof the Googlebot user-agent; scrapers do it constantly. So verification matters.
- Confirm it’s really Googlebot. Run a reverse DNS lookup on the requesting IP; it should resolve to a
googlebot.comorgoogle.comhost, and a forward lookup must match. Or check the IP against Google’s published crawler ranges. - Read Search Console Crawl Stats. Total requests, response codes, file types, and Googlebot type: your first stop when crawling looks off.
- Use URL Inspection. It shows the crawled status, the rendered HTML (vital for catching JS problems), and the canonical Google chose versus the one you declared.
- Analyze server logs. The ground truth. Logs reveal exactly which URLs Googlebot hits, how often, and what status it gets: the only way to spot crawl waste on a large site.
For the full sequence from fetch to index, see our guide to crawling, indexing, robots.txt, sitemaps and canonicals.
When Googlebot becomes a business problem
On a site of a few hundred pages that index normally, Googlebot needs nothing from you beyond a sane robots.txt and a sitemap. Search Console and URL Inspection are enough.
It costs money when the gap between “crawled” and “indexed” opens up. Crawl Stats show Googlebot spending its requests on filter and parameter URLs while new product or location pages wait weeks. Hundreds of pages sit in “Discovered, currently not indexed”. A JavaScript rebuild leaves URL Inspection showing an empty rendered page. A robots.txt change blocks a whole section by accident.
On a fractional SEO engagement, the two founders read the logs and Crawl Stats, find where crawl is wasted and fix it at the template level. When we rebuilt Prawomat, cutting 2,646 URLs to about 1,370 on click data was part of the work, and daily Google clicks reached 2.9x within 19 days.
Frequently Asked Questions
What is Googlebot in simple terms?
Googlebot is Google’s web crawler: an automated bot that visits web pages, reads their content, follows links to find more pages, and sends what it finds to Google’s index. Pages it can’t crawl or render won’t appear in Google Search, making Googlebot access the foundation of getting found.
How do I check if Googlebot is crawling my site?
Open Google Search Console and review Crawl Stats for request volume and response codes, or use URL Inspection on a specific page. For deeper detail, analyze server logs to see exactly which URLs Googlebot requests and what status codes it receives. Verify suspect IPs with reverse DNS.
Does Googlebot render JavaScript?
Yes. Googlebot uses an evergreen, headless Chromium engine to execute JavaScript and build the DOM in a second rendering stage after the initial HTML fetch. Rendering is queued and resource-heavy, so critical content and links served only via client-side JavaScript can be delayed or missed if rendering fails.
How is Googlebot different from Google-Extended?
Googlebot crawls and indexes pages for Google Search. Google-Extended is a robots.txt token with no crawler of its own that controls whether content Google crawls may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. AI features in Google Search, including AI Overviews, follow your Googlebot rules, so blocking Google-Extended does not affect Search or AI Overviews.
Can I block Googlebot from specific pages?
Yes, but pick the right tool. Use robots.txt Disallow to stop crawling, and a noindex meta robots tag or X-Robots-Tag header to remove a page from the index. Never use robots.txt alone to deindex: Google can’t see a noindex on a page it isn’t allowed to fetch, so a blocked page can still be indexed URL-only when other sites link to it.