Meta robots advanced settings are the per-page directives that tell search engines how to index a URL and how much of it to show in results — going well beyond the basic index/noindex toggle most SEO plugins expose by default. They cover snippet length, image and video previews, cached copies, expiry dates, and embedded-content rules. Get them right and you control exactly what Google and Bing index and display; get them wrong and you can quietly delist pages that drive revenue.
Meta Robots Advanced Settings
Meta robots advanced settings are the extended set of crawler directives — noarchive, nosnippet, noimageindex, notranslate, unavailable_after, indexifembedded, and the max-snippet, max-image-preview, and max-video-preview limits — that control how a search engine indexes and presents a page beyond the basic index/noindex and follow/nofollow rules.
What advanced meta robots actually controls
The basic robots meta tag handles two binary decisions: should this page be indexed, and should links on it be followed. The advanced directives split into two jobs that practitioners conflate at their peril:
- Access and indexing control — whether the page enters the index at all (
noindex), and whether equity flows through its links (nofollow). - Presentation control — how the page appears once indexed: snippet text, image thumbnails, video previews, cached copies, and translation offers.
That distinction matters because a noindex page can’t show a snippet anyway, while a nosnippet page still ranks — it just shows no description. Confusing the two is the single most common mistake we see in plugin configs. If you only need to suppress a SERP description, you don’t reach for noindex; you reach for nosnippet.
These directives live in one of two places. The HTML <meta name="robots"> tag in the document <head> works for HTML pages. The X-Robots-Tag HTTP response header does the same job for any resource — PDFs, images, feeds, JSON endpoints — where there’s no HTML head to inject a tag into. Same vocabulary, different delivery.
Rule of thumb: meta robots is for what Google does after it fetches a page. It is not a crawl blocker. If you want to stop a crawl entirely, that’s robots.txt — but a robots.txt-blocked URL can never read a
noindex, which is exactly how pages end up indexed without a snippet against your wishes.
The full directive vocabulary
Here’s the working set supported by Google, with current behavior — not the 2015 version still floating around in old tutorials.
| Directive | What it does | Notes |
|---|---|---|
index / noindex | Allow or block this page from the index | Default is index; you rarely need to declare it |
follow / nofollow | Follow or ignore links on the page | noindex, follow lets equity pass while keeping the page out of results |
noarchive | Suppress the cached-copy link | Google retired the public cache link in 2024, but the directive still governs cached use |
nosnippet | Show no text snippet or video preview | Also opts the page out of being used in AI Overviews and Gemini grounding |
max-snippet:[n] | Cap snippet length at n characters | max-snippet:0 is equivalent to nosnippet; -1 means no limit |
max-image-preview:[none|standard|large] | Control thumbnail size in results | large is effectively required to appear in many rich visual SERP features |
max-video-preview:[n] | Cap animated video preview to n seconds | 0 disables preview; -1 removes the limit |
noimageindex | Block images on the page from image search | Applies to images the page references, not the page itself |
notranslate | Suppress the “Translate this page” offer | Useful for region-locked or legal copy |
unavailable_after:[date] | Drop the page from the index after a date | RFC 822 / ISO 8601 date; good for expiring promos |
indexifembedded | Index a noindex page only when embedded | Pair with noindex for content meant to live inside iframes |
none | Shorthand for noindex, nofollow | Blunt instrument — be deliberate |
Two directives are dead and should be stripped from any config you inherit: noodp (DMOZ closed in 2017) and noydir (Yahoo Directory is gone). If a legacy plugin still emits them, it’s noise — harmless but a tell that the setup hasn’t been touched in years.
The AI Overviews reality
nosnippet and max-snippet now carry weight they didn’t a few years ago. Google has confirmed that content excluded via these directives is also excluded from being surfaced inside AI Overviews and other generative SERP features. So the call is genuinely strategic: suppress your snippet and you protect content, but you also opt out of the AI answer layer that increasingly sits above the blue links. For most pages you want maximum visibility there — which means not throttling snippets reflexively.
Combining directives correctly
Directives go in a single comma-separated value. You can also target a specific crawler by swapping the name attribute — googlebot, bingbot, or google-extended (which governs Gemini training, separate from Search indexing).
<!-- Keep this page out of results but still crawl its links -->
<meta name="robots" content="noindex, follow">
<!-- Index normally, large image preview, snippet capped at 160 chars -->
<meta name="robots" content="index, follow, max-image-preview:large, max-snippet:160">
<!-- Expire a promo page from the index after a set date -->
<meta name="robots" content="unavailable_after: 2026-12-31T23:59:59-05:00">
<!-- Bing-specific instruction without affecting Google -->
<meta name="bingbot" content="noindex">For non-HTML assets, the same logic moves to the header:
# Apache .htaccess — keep PDFs out of the index but allow image previews
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, noarchive, max-image-preview:standard"
</Files>When two directives conflict, the most restrictive one wins. If your template emits index but an inherited rule emits noindex, the page stays out. This is why a single mis-scoped global default in a CMS can silently delist a whole section — always check the rendered tag, not the plugin checkbox.
Where each directive earns its keep
Mapping directives to real page types is where this stops being trivia and starts being strategy. A clean baseline:
- Public, valuable content —
index, follow(the implicit default; leave it alone). - Faceted nav, internal search results, infinite filter combinations —
noindex, followso the crawler still discovers products but the junk URLs never index. This protects crawl budget on large sites where bots otherwise drown in parameter permutations. - Staging, admin, login, cart, account pages —
noindex, nofollowplus real authentication. Never rely on a robots directive as a security control. - Thin or near-duplicate category pages — usually a canonical is the better tool than
noindex, because canonical consolidates signals while keeping the page discoverable. - Expiring event or promo pages —
unavailable_after, then a 410 once the content is permanently gone. - Embeddable widgets and media meant for iframes —
noindex+indexifembedded.
The most damaging error in this whole topic: applying noindex site-wide by accident — a flipped global toggle, a staging config pushed to production, a theme update that resets defaults. We’ve watched sites lose their entire indexed footprint inside a week from one checkbox. If pages are silently dropping, the first thing to check is the rendered robots tag against your intent. For the WordPress-specific path, see how to noindex URLs in WordPress; for full removal workflows, how to unindex pages from search engines.
Verifying directives actually fired
Configuring a directive and confirming it took effect are two different things. The directive only works if a crawler can fetch the page and read the tag — which is why crawlability is a prerequisite. A robots.txt block prevents Googlebot from ever seeing the noindex, so the page can linger in the index marked “indexed, though blocked by robots.txt.”
Your verification checklist:
- Use the URL Inspection tool in Google Search Console to see the rendered robots directive Google actually read, not the source you intended.
- Check the HTTP header with a curl request (
curl -I) when you’re relying on X-Robots-Tag — headers don’t show up in page source. - Watch the index status in Search Console’s Pages report; a sudden jump in “Excluded by ‘noindex’ tag” is your early-warning system.
- Read your logs to confirm bots are fetching the URLs you expect, then re-crawling after a change. Understanding how web crawlers work tells you why a directive change can take days to propagate. Pages can sit as discovered, currently not indexed long after you’ve adjusted the tag.
Frequently Asked Questions
What is the difference between meta robots and X-Robots-Tag?
Meta robots is an HTML tag placed in the page <head>, so it only works on HTML pages. X-Robots-Tag delivers the identical directives through an HTTP response header, which lets you control non-HTML files like PDFs, images, and feeds. Both share the same directive vocabulary; you choose based on the resource type.
Does noindex stop Google from crawling a page?
No. noindex tells Google not to index a page, but the crawler still has to fetch the page to read that directive. If you also block the URL in robots.txt, Google can’t read the noindex at all, and the page may stay indexed without a snippet. To remove a page, allow crawling and let noindex apply.
Will nosnippet affect AI Overviews?
Yes. Google has confirmed that pages carrying nosnippet or max-snippet:0 are excluded from AI Overviews and generative answer features, in addition to losing their standard SERP snippet. For most pages you want visibility in the AI layer, so apply snippet-suppression directives deliberately and only where protecting content outweighs reach.
Is the noodp directive still useful?
No. noodp instructed engines not to pull titles and descriptions from the DMOZ Open Directory Project, which shut down in 2017. Google ignores it entirely now. If an inherited plugin or template still emits noodp or noydir, it’s harmless legacy clutter you can safely remove from your configuration.
How do I check which robots directive Google actually sees?
Use the URL Inspection tool in Google Search Console — it shows the rendered robots directive Google read, which can differ from your source if a tag is injected dynamically or inherited from a global rule. For X-Robots-Tag headers, run curl -I <url> since headers never appear in page source, and confirm the directive value matches your intent.
Meta robots advanced settings are where indexing strategy gets surgical — and where a single careless default can erase a site’s visibility. Treat every directive as a deliberate decision tied to a page type, verify the rendered output rather than trusting a checkbox, and keep the access-versus-presentation distinction front of mind. If you’re auditing a large or programmatic site where these rules multiply across templates, our AI SEO services cover exactly this kind of crawl-and-index control at scale. For adjacent foundations, see SEO site structure.