glossary

Crawlability Issues: 10 Common Problems and How to Fix Them

Crawlability is whether search bots can reach and read your pages. The 10 crawlability issues we find most in audits, how to check a page, and the fixes.

updated:

Crawlability is whether search engine bots can reach, fetch and read the pages on your site. A page is crawlable when a bot is allowed to request it, can find a link to it, gets a working response and can read the content and links that come back. Crawlability issues are whatever breaks one of those steps: a robots.txt block, server errors, links a bot cannot follow, orphan pages, redirect chains and content that only appears after JavaScript runs. Crawling sits one step before indexing and two before ranking, so a page that cannot be crawled gets no impressions, no clicks and no AI Overview citation. We see crawlability issues sink more good content than thin writing ever does, because the page never gets a fair hearing in the first place.

Crawlability

Crawlability is the degree to which search engine bots can discover, access, and read a site’s pages without technical barriers, governed by robots.txt, server responses, internal linking, render behavior, and sitemap signals.

Crawlability vs. indexability vs. rankability

The three get blurred constantly, and the confusion costs money. They are sequential gates: pass one to reach the next.

StageQuestion it answersControlled byCommon failure
CrawlabilityCan a bot reach and read the page?robots.txt, status codes, links, renderPage blocked or unreachable
IndexabilityShould this page be stored?noindex, canonicals, qualityPage crawled but dropped
RankabilityHow well does it compete?content, links, intent, E-E-A-TIndexed but invisible

Search Console’s two most confusing verdicts sit on either side of the first gate. “Discovered - currently not indexed” means Google knows the URL and has not crawled it yet, usually because it postponed the fetch; that is a crawl problem. “Crawled - currently not indexed” means Google fetched the page and chose not to store it; that is an indexability or quality problem. A page can be perfectly crawlable and still never index, and a well-written page can go uncrawled because nothing links to it. Fix the gates in order. Start with the mechanics in how do web crawlers work and the behavior of Googlebot.

How search engines crawl a site

A crawler starts from known URLs: your XML sitemap, pages it already knows, and links from other sites. From each page it fetches, it extracts links and queues new URLs. Google then renders the page, running its JavaScript, and reads the final DOM as well as the raw HTML.

Three things decide whether this loop reaches your important pages:

  • Permission: do robots.txt and your server allow the fetch?
  • Discoverability: is there a crawlable link path to the URL?
  • Comprehensibility: once fetched, can the bot read the content and follow its links?

Break any one and the page falls out of the loop.

Field note: the most expensive crawlability bug is a silent one. A Disallow: line that has sat in robots.txt for two years, fencing off a section nobody remembers blocking. Nothing breaks. The pages just never appear.

Common crawlability issues (and how to fix them)

These are the crawlability problems we hit on real audits, roughly in order of how often they cause damage.

  1. robots.txt blocking: an over-broad Disallow fences off pages or whole directories. Fix: read the live file at /robots.txt, check Search Console’s robots.txt report (under Settings) for fetch errors, test individual URLs with URL Inspection, and narrow rules to only what truly belongs blocked. Watch the file’s own status code too: if /robots.txt answers with a 5xx error, Google treats the whole site as disallowed until it can fetch the file again.
  2. noindex where you wanted visibility: a meta robots or X-Robots-Tag noindex leaks onto pages that should rank, often from a template default or a staging config shipped to production. Strictly an indexability bug, it travels with crawl problems so often that it belongs on this list. Fix: audit the meta robots settings and strip noindex from anything you want indexed.
  3. Server errors and timeouts: repeated 5xx responses, 429s or slow replies make Googlebot slow its crawling down. Fix: resolve the errors and tighten response times; see how site speed influences SEO.
  4. JavaScript-dependent content: critical content or links that only exist after client-side rendering can be missed or picked up late. Fix: server-render or pre-render key content and build navigation from real <a href> links. Start with making your JavaScript SEO-friendly.
  5. Orphan pages: URLs with zero internal links pointing in. A sitemap can get them discovered, but linked pages get crawled far more reliably. Fix: link to them from relevant hubs; see orphaned content.
  6. Deep architecture: pages buried five or more clicks from the homepage tend to be crawled less often. Fix: flatten the structure and improve hub linking per SEO site structure and website architecture.
  7. Redirect chains and loops: every hop costs a fetch, and Googlebot gives up after 10 hops. Fix: point each old URL straight at its final destination with a single 301 and break any loops; see redirect chains.
  8. Broken links and soft 404s: true 404s and pages that return 200 with “not found” content both waste crawl activity. Fix: repair links and return honest status codes; read soft 404 errors and the basics of the 404 status.
  9. Sitemap drift: a sitemap full of redirected, 404 or noindex URLs sends crawlers to pages that will never count and makes the file useless as a check on what you want indexed. Fix: keep it to canonical, 200-status, indexable URLs and submit it to search engines.
  10. Crawl waste from low-value URLs: faceted filters and session parameters spawn thousands of variants that soak up attention. Fix: canonicalize or block the junk so crawl budget lands on pages that matter.

Google follows a link only when it is an <a> element with an href attribute that holds a resolvable URL. Anything else may work for a visitor clicking around and still be invisible to the crawler.

Link markupCrawlable?
<a href="https://example.com/shoes/">Yes
<a href="/shoes/"> (relative URL)Yes
<a href="/shoes/" onclick="track()">Yes, because the href holds a real URL
<a onclick="goTo('/shoes/')">No, there is no href
<a href="javascript:goTo('shoes')">No, the href is a script
<span href="/shoes/"> or a clickable <div>No, it is not an <a> element
<a routerLink="shoes"> (framework attribute)No, unless the framework also renders an href

A nofollow link is crawlable in the mechanical sense. It asks search engines not to pass credit through it, and Google treats that as a hint; see nofollow links. For image links, wrap the <img> in an <a href> and give it alt text so the link has anchor text.

How to check if a page is crawlable

Do not guess. Pull the evidence in this order:

  1. URL Inspection in Search Console. Run “Test live URL” on the page. It reports whether crawling is allowed, whether the fetch succeeded, and, under “View tested page”, the rendered HTML and a screenshot. This is the replacement for the old “Fetch and Render” (Fetch as Google) tool, which no longer exists. Cross-check with how to see when Google last crawled a page.
  2. The Pages report. It shows indexing verdicts across the site. A growing “Discovered - currently not indexed” count is the crawl-side warning; read discovered, currently not indexed for what each verdict means.
  3. The Crawl Stats report (under Settings). Watch for response-code spikes (5xx, 4xx), host availability warnings and falling crawl volume, the early signs of a server or robots problem. It also breaks requests down by Googlebot type.
  4. A full-site crawl. Run Screaming Frog or Sitebulb to surface orphan pages, redirect chains, blocked URLs and click depth at scale, then compare the result against your sitemap.
  5. Server logs. The only source that shows what every bot actually requested, and the way to confirm whether bots reach a section at all. Anyone can claim to be Googlebot in a user-agent string, so verify real Googlebot hits with a reverse DNS lookup or against the IP ranges Google publishes.

For a deeper walk through robots.txt, sitemaps and canonicals, see our guide to crawling and indexing. This is also the workflow inside our core programmatic SEO audits, where one template-level crawl bug can hide thousands of pages at once.

Crawlability for AI crawlers

Google’s AI Overviews and AI Mode draw on Google’s own index, so a page Googlebot cannot crawl cannot be cited there either. The other answer engines send their own crawlers, such as OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot and PerplexityBot, and each of them reads your robots.txt. That makes the file a control over which answer engines can read your content, alongside ordinary search.

Google-Extended works differently. It is a robots.txt control token with no crawler of its own: Googlebot does the fetching, and the token decides whether content Google crawls may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Disallowing it has no effect on Google Search, AI Overviews or rankings.

To see where your file stands, enter your domain or paste your robots.txt into our free AI-Bot Access Checker. It reports whether GPTBot, ClaudeBot, PerplexityBot and the other AI bots on its list may crawl the site, and what your Google-Extended rule allows.

When crawlability becomes a business problem

A few hundred pages, all linked from the navigation and listed in a clean sitemap? URL Inspection and a free crawler find almost everything, and an afternoon fixes it.

It costs real money when the problem is systematic. A whole section is missing from Search Console. “Discovered - currently not indexed” climbs into the thousands. New pages take weeks to get crawled. A framework release drops links from every template, or a migration ships a staging robots rule. Those are template and infrastructure problems.

On a fractional SEO engagement, the two founders find the rule or template behind it, fix it once and confirm in the logs that bots reach the pages. When we rebuilt Prawomat as static pages with zero scripts on a letter page (one release, alongside rewritten content and pruned URLs), daily Google clicks rose 2.9x in 19 days, against 1.3x on the same dates in 2025.

Frequently Asked Questions

What are crawlability issues?

Crawlability issues are technical barriers that stop search bots from reaching or reading your pages: robots.txt blocks, server 5xx errors, links built without an href, redirect chains, orphan pages and JavaScript-only content. Each one prevents a URL from being fetched, understood or queued for indexing, so the page never reaches search results. An accidental noindex tag is usually listed with them, though it blocks indexing after the crawl.

What does crawlable mean?

A crawlable page or link is one a search engine bot can find, is allowed to fetch, and can read. For a link, it means an <a> element with an href that holds a real URL. For a page, it means robots.txt allows it, the server returns it with a 200 status, and its content and links are present in the HTML or in the rendered page.

How do I check if my page is crawlable?

Use the URL Inspection tool in Google Search Console and run “Test live URL”. It shows whether crawling is allowed, whether Google could fetch and render the page, and what the rendered HTML looks like. Then check the Pages report for indexing verdicts and run a crawler such as Screaming Frog to catch orphan pages, blocked paths and redirect chains across the site.

What’s the difference between crawlability and indexability?

Crawlability is whether a bot can reach and read a page; indexability is whether the engine then decides to store it. A page must be crawlable before it can be indexed, and a crawlable page can still be excluded by a noindex tag, a canonical pointing elsewhere or a quality judgment. They are sequential gates.

Does robots.txt affect crawlability?

Yes: robots.txt is the first gate. A Disallow rule tells crawlers not to fetch matching URLs, so their content cannot be read. A blocked URL can still be indexed as a bare link if other pages point to it, and robots.txt cannot remove a page from the index. For that, use a noindex tag and leave the page crawlable, because Google has to fetch the page to see the tag.

Can JavaScript hurt crawlability?

It can. If critical content or links only appear after client-side rendering, crawlers may miss them or pick them up late, and click handlers without an href are never followed as links. Server-rendering or pre-rendering key content, and building navigation from real <a href> links, keeps JavaScript-heavy pages fully crawlable.

Can I see which bots crawl my site?

Yes. Your server logs record every request with its user-agent, so you can see Googlebot, Bingbot, GPTBot, ClaudeBot and the rest, page by page. User-agents can be faked, so confirm the ones that matter with a reverse DNS lookup or the operator’s published IP ranges. For Google alone, the Crawl Stats report in Search Console summarizes requests by Googlebot type.

where we come in

Whole sections of your site missing from Google, and no idea why?

Senior SEO without hiring a team: the two founders run strategy and execution for a fixed monthly fee, one client per vertical.

2.9x

Prawomat: Daily Google clicks; 1.3x on the same dates in 2025

read the case study →

or ask a founder

30 min · intro, founder-to-founder

Book a call