Technical SEO Audit from the Server Up: CWV, Log Files, and Crawl Budget

I’m a psychometrician, not an SEO guy. But the thing about psychometrics is that every configuration is a hypothesis, and a hypothesis without data is just an opinion. When a site isn’t ranking despite “doing the SEO things,” the correct response isn’t to do more things. It’s to run a proper technical SEO audit and find out what’s actually broken.

That’s the discipline this post is about. Not how crawling works or how to implement structured data; the Batch 2 crawling and technical-SEO posts cover that implementation layer. This is the diagnostic layer: your site is live, something is underperforming, and you need a methodology for tracing the problem from the server outward. Log files first, then Core Web Vitals, then crawl budget, then the server signals that compound everything else.

The audit-before-fix discipline is the whole point. Most technical SEO work targets symptoms because practitioners skip the diagnosis: they read that LCP needs to be under 2.5 seconds, add a lazy-loading plugin, gain 0.2 seconds, while the real problem, a misconfigured cache serving PHP-rendered HTML on every Googlebot request, never gets touched. The server knows what’s really happening. You just have to read it.

Start at the Server: What Log Files Tell You That GSC Doesn’t

Google Search Console is a downstream signal: what Google’s systems recorded after processing your site. Server logs tell you what actually happened on the wire: every request, every response code, every bot visit, every real user hit, timestamped and sequential. The gap between those two data sources is where the interesting problems live.

What server logs contain

A typical Nginx access log line shows the requesting IP, timestamp, HTTP method and URL, response status code, bytes transferred, referrer, and user agent. The user agent is the tell: Googlebot, Bingbot, real users, and scanner bots all announce themselves differently.

The SEO-relevant signals to extract from logs:

Googlebot request frequency: how often is the crawler visiting, and is that frequency consistent or erratic? A drop in Googlebot visits often precedes a coverage drop in GSC by days or weeks.
Response time distribution: what’s the 50th, 95th, and 99th percentile TTFB for bot requests vs real users? If Googlebot is seeing 800ms TTFB where real users see 50ms, you have a cache configuration problem.
5xx error rate: any 500-level responses to Googlebot are crawl failures. Even a sub-1% rate compounds over time.
Redirect chain length: a bot that follows three 301s to reach a URL has burned crawl budget on redirects, not content.
Cache hit rate: this is the one most operators miss.

The cache hit rate signal

On a properly configured LEMP stack, Nginx’s FastCGI cache should be serving cached HTML for the vast majority of requests. A cache hit means Nginx reads a file from disk (sub-50ms). A cache miss means the request passes through to PHP-FPM, executes WordPress, queries MariaDB, and builds the response, typically 300-800ms on a tuned stack, longer on an untuned one.

Add a custom log variable that records whether each response was a cache HIT, MISS, or BYPASS, then calculate your cache hit rate for Googlebot requests specifically. I’ve seen stacks with a 12% cache hit rate where the operator assumed it was 90%+; a single misconfigured bypass rule was sending Googlebot to PHP on every request. That means Googlebot sees your slow origin, which directly affects how it allocates crawl budget and how fast your pages get indexed. I covered the four-layer cache architecture that produces the numbers you want to see here in the WordPress self-hosting guide. That’s the infrastructure build reference. The audit tells you whether the build is actually working.

How to read logs without a log aggregation stack

You don’t need Elasticsearch and Kibana for a first-pass audit. Screaming Frog Log File Analyser handles a single site: import logs, segment by bot, filter by URL pattern, get response-time distributions. For quick extractions, basic grep and awk get you far:

# Count Googlebot hits by status code
grep -i googlebot /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# Find URLs returning 5xx to any user agent
awk '$9 ~ /^5/' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn

The goal at this stage isn’t a pretty dashboard. It’s the anomalies: a high 5xx rate, a surprising 200 response for a page you thought was 404, a bot hammering a URL that shouldn’t exist.

Core Web Vitals: Measuring What Google Actually Measures

Core Web Vitals are Google’s standardized, user-experience-grounded ranking signals. Three metrics, each targeting a different failure mode in the user experience:

LCP (Largest Contentful Paint): how long until the largest visible content element renders. On most sites that’s a hero image or above-the-fold heading. The passing threshold is under 2.5 seconds. In practice it fails on hero images served without width/height attributes, images not preloaded when they’re the LCP candidate, large images not served in next-gen formats (WebP, AVIF) or via CDN edge, and slow TTFB pushing everything downstream.

INP (Interaction to Next Paint): this replaced FID (First Input Delay) as the interactivity metric in March 2024. Where FID measured the delay before the browser started processing the first interaction, INP measures the full end-to-end latency of all interactions throughout the page lifecycle, not just the first one. Passing threshold: under 200ms. It fails on third-party scripts (analytics, live chat, marketing pixels) executing blocking JavaScript on the main thread, WordPress plugins that inject large scripts globally rather than conditionally, and large DOM trees forcing expensive layout recalculations.

CLS (Cumulative Layout Shift): how much the page’s content moves unexpectedly after initial render. Passing threshold: below 0.1. It fails on <img> tags without explicit width and height (the browser doesn’t know the space to reserve), web fonts swapping after initial render (add font-display: swap but also preload the fonts), and ad slots or embeds that resize after content renders above the fold.

Field data vs lab data: the distinction that matters

PageSpeed Insights and Lighthouse run lab data: a synthetic measurement made by one tool, once, on a specific connection. The Chrome User Experience Report (CrUX) is field data: aggregated real-user measurements across actual Chrome users visiting your site.

Google uses field data for ranking. When CrUX data exists for your URL, it feeds the Core Web Vitals report in GSC; when it doesn’t (typically low-traffic pages), Google falls back to site-wide or origin-level data. The practical implication: your PageSpeed score and your actual ranking signal can diverge. A site can post a 95 Lighthouse score and still fail CWV field data because the real-user device distribution includes low-end Android phones where INP fails consistently, and a synthetic desktop test doesn’t catch it.

The audit workflow: GSC Core Web Vitals report → identify failing URLs and their failure type → PageSpeed Insights per-URL to get the diagnostic detail → trace root cause.

The TTFB connection

CWV performance lives downstream of server speed. LCP cannot pass if the server takes 1.2 seconds to send the first byte; the browser hasn’t received the content to render yet. So a meaningful CWV audit starts at origin performance, not browser performance. TTFB (Time to First Byte) is driven by server processing time (PHP execution plus database queries on dynamic requests), network latency (reduced by CDN edge caching), and connection setup (HTTP/2 multiplexing, TLS session resumption). Fix the origin before chasing browser-side metrics. The relationship between site speed and SEO is upstream of every CWV optimization.

The remediation actions for each failure mode (image optimization, script deferral, font loading strategy) are covered in the how to increase website speed reference. The audit tells you which failure type you have; that guide tells you the fix sequence.

Crawl Budget: Diagnosing Waste and Fixing It

Crawl budget is Google’s finite allocation for how many of your pages it crawls per unit of time. For most sites under 1,000 pages it isn’t a concern: Google crawls everything important. It becomes a real constraint on large sites (1,000+ indexable URLs), sites with lots of near-duplicate content, and (this one catches people) programmatic SEO programs that generate pages faster than they get crawled.

When budget is wasted, your best pages get crawled less often, indexing slows, and freshness signals degrade. The audit goal is to find where budget leaks and stop it.

Where crawl budget gets wasted

Infinite URL spaces: Session ID parameters (?sessionid=abc123) create unique URLs from the crawler’s perspective even though the content is identical. Sort and filter parameters are the classic case: /shop/?color=red&size=M&sort=price-asc generates a URL-space explosion unless these are canonicalized or parameterized away.

Soft 404s: Pages returning a 200 status code that contain “not found” or “no results” content. The server says “this page exists,” Googlebot crawls it, and finds nothing worth indexing. Common examples: empty search results, expired event pages, discontinued product pages.

Redirect chains: A URL that redirects to a URL that redirects again before reaching the destination. Each hop burns time and degrades link equity. Log file analysis exposes these: look for 301 or 302 responses where the target URL is itself in the redirect log.

Thin-content pages: Pages with low informational value that consume crawl budget without contributing indexable signal. Google’s systems deprioritize crawling pages they’ve learned have low quality, which means if your thin pages get crawled, your good pages get crawled less.

Orphaned pages: URLs with no internal links pointing to them. Crawlability depends on the internal link graph as much as sitemaps. A page with zero inbound internal links exists in a crawl dead-zone.

The crawl budget audit workflow

The cross-reference is the key move here:

Extract Googlebot-visited URLs from server logs (last 30-90 days)
Pull the URL list from GSC Coverage report, particularly the “Discovered, currently not indexed” and “Crawled, currently not indexed” segments
Cross-reference: which URLs is Googlebot visiting that are NOT in your “indexed” set?
For each non-indexed crawled URL: trace the reason (quality signal, canonical mismatch, thin content, server speed, or discovered but not yet indexed)

That last category (“Discovered, currently not indexed”) is one of GSC’s most diagnostic signals. It means Googlebot found the URL (via internal links or a sitemap) but hasn’t crawled it. If this pool is large and growing, you’re generating indexable URLs faster than crawl budget can process them, or you have a server-level crawlability problem.

Crawl budget and programmatic SEO

Large pSEO programs have a specific failure mode: thousands of new pages pull Googlebot’s crawl time toward fresh URLs, starving high-performing existing pages of re-crawl frequency. Consolidate non-performing pages via canonicals or noindex so the budget concentrates on pages that earn their indexing. And treat the XML sitemap as a budget management tool, not just a discovery tool: include only URLs you want indexed, and set lastmod to reflect actual content changes; a lastmod that always shows today’s date teaches Googlebot the signal is noise.

Server Signals That Compound Technical SEO

The log and CWV audit surfaces symptoms. These server-level signals are the underlying conditions that generate them.

HTTPS end-to-end

HTTPS is a confirmed ranking signal, and the detail that causes silent failures is the end-to-end chain. Cloudflare’s “Full (strict)” mode requires a valid origin certificate between Cloudflare and your server, in addition to the certificate Cloudflare presents to visitors. A site running “Flexible” mode (Cloudflare-to-origin over HTTP) looks encrypted to users but transmits content unencrypted on the back half. Verify your Cloudflare SSL mode, the origin cert expiry, and that the Cloudflare-to-origin connection actually completes TLS.

HTTP/2 and HTTP/3

HTTP/2’s connection multiplexing reduces overhead for Googlebot when crawling multiple resources from the same origin; it requests them over a single TCP connection rather than opening a new one per resource. HTTP/3 (over QUIC) reduces latency further on variable-quality connections. Verify both at the server level (curl -I --http2 https://yourdomain.com) and the Cloudflare layer; most modern CDN setups activate HTTP/3 by default, but confirm it isn’t disabled.

Response code auditing in bulk

How a crawler reaches your pages is shaped by your redirect map, and a bulk crawl (Screaming Frog) with response-code filtering exposes three patterns the incremental-development view hides:

301 chains: a URL redirects to another URL that was also redirected, producing multi-hop chains. Every 301 in production should go directly to its final destination; every chain of more than one hop is a remediation item, and the impact compounds on sites with a history of domain migrations or URL restructures.

Accidental 302s: a 302 (temporary) redirect tells Googlebot not to transfer link equity and not to update its index, because the redirect is “temporary.” Framework and CMS defaults sometimes emit 302 where 301 is correct, so old URLs stay in the index.

Soft 404s at scale: 200-OK pages that contain “not found” phrases. Cross-reference against the server log to see if Googlebot is wasting crawls on them.

Mobile-first indexing at the server level

Google indexes from the mobile user agent (Googlebot-Mobile). Segment your log analysis by user agent and verify both Googlebot and Googlebot-Mobile requests get the same response codes, same content, and same TTFB. A mobile-specific caching rule, redirect, or server-side user-agent detection that produces divergent responses is a mobile-first indexing issue that won’t surface in a desktop crawl.

Building an Audit Checklist You’ll Actually Run

An audit isn’t an event. It’s a recurring cadence. The site changes, Google’s systems evolve, the crawl pattern shifts, and a configuration that was correct six months ago has drifted. The checklist is the mechanism that keeps you diagnostic over time, not just at launch.

Three-tier cadence

Monthly (server and logs):

Pull 30-day log sample; segment by Googlebot/real users
Calculate cache hit rate for Googlebot requests
Check 5xx rate for bot traffic
Review TTFB distribution (50th/95th percentile by bot vs user)
Spot-check redirect chains for newly added URLs
Verify HTTPS chain end-to-end (origin cert expiry)

Weekly (CWV and GSC):

GSC Core Web Vitals report: any new failing URLs?
GSC Coverage: change in “Discovered, currently not indexed” volume?
GSC Coverage: any increase in “Crawled, currently not indexed”?
PageSpeed Insights on any newly published page types
INP check on any pages with new interactive elements

Quarterly (full crawl):

Full site crawl (Screaming Frog or equivalent): identify broken links, redirect chains, missing meta, duplicate titles/descriptions
Soft 404 audit: 200-OK pages with no-content/error messaging
Internal link audit: orphaned pages with zero inbound links
Crawlability audit: compare sitemap to crawled URL set; identify submitted URLs that Googlebot hasn’t visited
XML sitemap accuracy: remove non-indexable URLs, verify lastmod values

Audit-to-action: turning findings into a fix list

The audit produces findings; the value is in prioritization. Weight each finding by severity × effort × impact. Correctness failures (5xx errors, HTTPS failures, mobile-indexing divergence, soft 404s at scale) get immediate attention regardless of effort. Everything else is weighted by impact: a three-hop redirect chain affecting 200 pages is worth fixing now; one affecting three pages can wait, and minor CLS scores or image-format tweaks get batched. Run this assessment explicitly rather than fixing whatever’s at the top of the list; the most visible finding is often not the highest-impact one.

What to automate vs what needs judgment

Automate detection: scheduled crawls, log aggregation, GSC data pulls via API, uptime monitoring. That’s measurement, and measurement should run continuously. Reserve human judgment for interpretation and prioritization: which finding is blocking the most valuable pages, and what a given pattern actually means.

The audit is a measurement instrument: it tells you what’s happening, not what it means. That’s the domain experience part.

Run the Audit or Delegate It

A proper first-pass technical SEO audit (logs, CWV, crawl budget, server signals) takes 4-8 focused hours; recurring audits run in 1-2 once the infrastructure is set up. The interpretation layer on top takes experience across enough sites to know when “discovered, not indexed” is a temporary crawl-queue lag versus a structural quality problem.

If you want to run this yourself, the checklist above is the methodology. If you want it as a structured deliverable with findings, prioritization, and a fix plan, that’s what our fractional SEO service covers: technical audit included, ongoing diagnostic cadence built in.

Frequently Asked Questions

What is a technical SEO audit and what does it cover?

A technical SEO audit is a diagnostic pass over a site’s infrastructure, server configuration, and crawl data to identify barriers to indexing and ranking. It covers server logs, Core Web Vitals, crawl budget efficiency, response codes, redirect chains, HTTPS configuration, and mobile-first indexing signals, distinct from on-page or content audits.

How do server logs help with SEO?

Server logs record every request Googlebot makes to your site, including response codes, TTFB per request, cache hit or miss status, and frequency of visits by URL. They surface problems that GSC doesn’t report (like a cache misconfiguration causing Googlebot to hit slow PHP-rendered responses) before those problems appear in ranking data.

What Core Web Vitals scores does Google use for ranking?

Google uses field data from the Chrome User Experience Report (CrUX), meaning real user measurements, not synthetic lab data from PageSpeed Insights or Lighthouse. The three metrics are LCP (Largest Contentful Paint, threshold: under 2.5s), INP (Interaction to Next Paint, threshold: under 200ms; INP replaced FID in March 2024), and CLS (Cumulative Layout Shift, threshold: below 0.1).

How do you identify crawl budget waste?

Cross-reference your server logs (which URLs Googlebot visited) against the GSC Coverage report (which URLs are indexed). URLs Googlebot crawls repeatedly without indexing signal wasted budget; trace each cause: soft 404, thin content, canonical mismatch, or server speed. Infinite URL spaces and redirect chains are the most common structural sources.

How often should you run a technical SEO audit?

Run server and log review monthly; it catches cache degradation and 5xx spikes before they compound. Run GSC and CWV monitoring weekly, since those signals update frequently. Run a full site crawl (broken links, redirect chains, soft 404s, orphans) quarterly, or after any significant change like a URL restructure or CMS migration.