Meta robots advanced settings are the per-page directives that tell search engines how to index a URL and how much of it to show in results, going well beyond the basic index/noindex toggle. In Yoast SEO the field is literally called “Meta robots advanced” and offers three options: No Image Index, No Archive and No Snippet. The full vocabulary covers snippet length, image and video previews, cached copies, expiry dates, and embedded-content rules. Get them right and you control exactly what Google and Bing index and display; get them wrong and you can quietly delist pages that drive revenue.
Meta Robots Advanced Settings
Meta robots advanced settings are the extended set of crawler directives (noarchive, nosnippet, noimageindex, notranslate, unavailable_after, indexifembedded, and the max-snippet, max-image-preview, and max-video-preview limits) that control how a search engine indexes and presents a page beyond the basic index/noindex and follow/nofollow rules.
Meta robots advanced in Yoast and other SEO plugins
Most people meet these settings in a WordPress SEO plugin. In Yoast SEO, open a post or page, go to the Advanced tab of the Yoast sidebar, and you will find Meta robots advanced with three options:
| Yoast option | Directive it adds | What it does |
|---|---|---|
| No Image Index | noimageindex | Asks Google not to index the images on this page. An image that also appears on another page without the rule can still be indexed. |
| No Archive | noarchive | Asks search engines not to keep a cached copy. Google no longer uses it; Bing still does (see the table below). |
| No Snippet | nosnippet | Shows no text snippet in results, and keeps the text out of Google’s AI Overviews and AI Mode as a direct input. |
Leave the field empty and the page follows your site-wide defaults, which is the right answer for almost every page. Rank Math and All in One SEO offer similar per-post checkboxes, usually with the max- limits as well.
What the default tag means
Out of the box, Yoast prints this on every public page, and many other plugins and CMS extensions print something close to it:
<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1">Read left to right: index this page, follow its links, no limit on snippet length, allow large image previews, no limit on video preview length. Every value is real and documented by Google, and together they are the most permissive settings available, which is what you want on a page you want found. max-image-preview:large matters most: Google Discover recommends large images, and this value is what permits them. If you see all instead (or x-robots-tag: all), it means no restrictions, which is already the default.
So the recommended setting for a normal public page is the default. Change it only for a specific page type with a specific reason, from the plugin’s settings or its filters. Editing the theme to strip the tag usually comes back with the next plugin update.
What advanced meta robots actually controls
The basic robots meta tag handles two binary decisions: should this page be indexed, and should links on it be followed. The advanced directives split into two jobs that practitioners conflate at their peril:
- Access and indexing control: whether the page enters the index at all (
noindex), and whether equity flows through its links (nofollow). - Presentation control: how the page appears once indexed: snippet text, image thumbnails, video previews, cached copies, and translation offers.
That distinction matters because a noindex page can’t show a snippet anyway, while a nosnippet page still ranks; it just shows no description. Confusing the two is a common mistake in plugin configs. If you only need to suppress a SERP description, you don’t reach for noindex; you reach for nosnippet.
These directives live in one of two places. The HTML <meta name="robots"> tag in the document <head> works for HTML pages. The X-Robots-Tag HTTP response header does the same job for any resource (PDFs, images, feeds, JSON endpoints) where there’s no HTML head to inject a tag into. Same vocabulary, different delivery.
Rule of thumb: meta robots is for what Google does after it fetches a page. It is not a crawl blocker. If you want to stop a crawl entirely, that’s robots.txt. But a robots.txt-blocked URL can never read a
noindex, which is exactly how pages end up indexed without a snippet against your wishes. Our free robots.txt generator builds the crawl rules side and keeps the two jobs apart.
The full directive vocabulary
Here’s the working set supported by Google, with current behavior. Plenty of old tutorials still describe the 2015 version.
| Directive | What it does | Notes |
|---|---|---|
index / noindex | Allow or block this page from the index | Default is index; you rarely need to declare it |
all | No restrictions | The default; listing it changes nothing |
follow / nofollow | Follow or ignore links on the page | noindex, follow keeps the page out of results while its links stay crawlable. Google’s John Mueller has said a page left on noindex for a long time tends to end up treated like noindex, nofollow |
noarchive | Suppress the cached copy | Google lists it as no longer used, since its cached links are gone. Bing still reads it: since 2023 it keeps the page out of Bing’s chat answers (now Copilot) and out of training for its generative models |
nosnippet | Show no text snippet or video preview | Also keeps the content from being used as a direct input for AI Overviews and AI Mode |
max-snippet:[n] | Cap snippet length at n characters | max-snippet:0 is equivalent to nosnippet; -1 means no limit |
max-image-preview:[none|standard|large] | Control thumbnail size in results | Applies to web search, Images and Discover; Discover recommends large images, which need large |
max-video-preview:[n] | Cap animated video preview to n seconds | 0 disables preview; -1 removes the limit |
noimageindex | Block images on the page from image search | Applies to the images the page references; the page itself stays indexable |
notranslate | Suppress the “Translate this page” offer | Useful for region-locked or legal copy |
unavailable_after:[date] | Drop the page from the index after a date | Any widely used date format (RFC 822, RFC 850, ISO 8601); ignored without a valid date. Good for expiring promos |
indexifembedded | Index a noindex page only when embedded | Pair with noindex for content meant to live inside iframes |
none | Shorthand for noindex, nofollow | Blunt instrument: be deliberate |
Two directives are dead and should be stripped from any config you inherit: noodp (DMOZ closed in 2017) and noydir (Yahoo Directory is gone). If a legacy plugin still emits them, it’s noise: harmless but a tell that the setup hasn’t been touched in years. A header such as x-robots-tag: all, noodp therefore reduces to plain all, which means no restrictions.
The AI Overviews reality
nosnippet and max-snippet now carry weight they didn’t a few years ago. Google has confirmed that content excluded via these directives is also excluded from being surfaced inside AI Overviews and other generative SERP features. So the call is genuinely strategic: suppress your snippet and you protect content, but you also opt out of the AI answer layer that increasingly sits above the blue links. For most pages you want maximum visibility there, which means not throttling snippets reflexively.
Combining directives correctly
Directives go in a single comma-separated value. You can also target a specific crawler by swapping the name attribute. Google supports two names there, googlebot and googlebot-news, and ignores any other value; Bing supports bingbot. Google-Extended is a robots.txt token only, and Google ignores a google-extended meta tag: to keep content out of Gemini training and grounding, disallow Google-Extended in robots.txt.
<!-- Keep this page out of results but still crawl its links -->
<meta name="robots" content="noindex, follow">
<!-- Index normally, large image preview, snippet capped at 160 chars -->
<meta name="robots" content="index, follow, max-image-preview:large, max-snippet:160">
<!-- Expire a promo page from the index after a set date -->
<meta name="robots" content="unavailable_after: 2026-12-31T23:59:59-05:00">
<!-- Bing-specific instruction without affecting Google -->
<meta name="bingbot" content="noindex">For non-HTML assets, the same logic moves to the header:
# Apache .htaccess: keep PDFs out of the index but allow image previews
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, noarchive, max-image-preview:standard"
</Files>When two directives conflict, the most restrictive one wins. If your template emits index but an inherited rule emits noindex, the page stays out. This is why a single mis-scoped global default in a CMS can silently delist a whole section. Always check the rendered tag: the plugin checkbox only tells you what you asked for.
Where each directive earns its keep
Mapping directives to real page types is where this stops being trivia and starts being strategy. A clean baseline:
- Public, valuable content:
index, follow(the implicit default; leave it alone). - Faceted nav, internal search results, infinite filter combinations:
noindex, followso the crawler still discovers products but the junk URLs never index. Note what this does not fix: Google still has to fetch every one of those URLs to read the tag. If bots are drowning in parameter permutations and wasting crawl budget, the fix is a robots.txt disallow on the patterns, applied once they have dropped out of the index. - Staging, admin, login, cart, account pages:
noindex, nofollowplus real authentication. Never rely on a robots directive as a security control. - Thin or near-duplicate category pages: usually a canonical is the better tool than
noindex, because canonical consolidates signals while keeping the page discoverable. - Expiring event or promo pages:
unavailable_after, then a 410 once the content is permanently gone. - Embeddable widgets and media meant for iframes:
noindex+indexifembedded.
The most damaging error in this whole topic: applying noindex site-wide by accident: a flipped global toggle, a staging config pushed to production, a theme update that resets defaults. One checkbox can take a site’s entire indexed footprint with it within days. If pages are silently dropping, the first thing to check is the rendered robots tag against your intent. For full removal workflows, including the WordPress-specific path, see how to unindex pages from search engines.
Verifying directives actually fired
Configuring a directive and confirming it took effect are two different things. The directive only works if a crawler can fetch the page and read the tag, which is why crawlability is a prerequisite. A robots.txt block prevents Googlebot from ever seeing the noindex, so the page can linger in the index marked “indexed, though blocked by robots.txt.”
Your verification checklist:
- Use the URL Inspection tool in Google Search Console to see the robots directive Google actually read in the rendered page.
- Check the HTTP header with a curl request (
curl -I) when you’re relying on X-Robots-Tag, since headers don’t show up in page source. - Watch the index status in Search Console’s Page indexing report; a sudden jump in “Excluded by ‘noindex’ tag” is your early-warning system.
- Read your logs to confirm bots are fetching the URLs you expect, then re-crawling after a change. Understanding how web crawlers work tells you why a directive change can take days to propagate. Pages can sit as discovered, currently not indexed long after you’ve adjusted the tag.
When meta robots becomes a business problem
One page with the wrong directive is a two-minute fix in the plugin sidebar. If you run a blog or a brochure site on default Yoast settings, leave them alone: they are already right.
It gets expensive when a directive sits on a template or a site-wide default. A staging noindex ships with a launch. A theme update resets the global setting. A category template inherits noindex and takes 3,000 product pages with it. An X-Robots-Tag rule on the server or CDN catches HTML it was never meant for. Search Console shows the damage as a jump in “Excluded by ‘noindex’ tag”, usually after traffic has already fallen.
On a fractional SEO engagement, the two founders map every page type to its intended directives, check what each release actually ships, and watch the index report after every change.
Frequently Asked Questions
What is the difference between meta robots and X-Robots-Tag?
Meta robots is an HTML tag placed in the page <head>, so it only works on HTML pages. X-Robots-Tag delivers the identical directives through an HTTP response header, which lets you control non-HTML files like PDFs, images, and feeds. Both share the same directive vocabulary; you choose based on the resource type.
Does noindex stop Google from crawling a page?
No. noindex tells Google not to index a page, but the crawler still has to fetch the page to read that directive. If you also block the URL in robots.txt, Google can’t read the noindex at all, and the page may stay indexed without a snippet. To remove a page, allow crawling and let noindex apply.
Will nosnippet affect AI Overviews?
Yes. Google has confirmed that pages carrying nosnippet or max-snippet:0 are excluded from AI Overviews and generative answer features, in addition to losing their standard SERP snippet. For most pages you want visibility in the AI layer, so apply snippet-suppression directives deliberately and only where protecting content outweighs reach.
What do No Image Index, No Archive and No Snippet do in Yoast?
They are the three options in Yoast’s “Meta robots advanced” field. No Image Index adds noimageindex, which asks Google not to index the images on that page. No Archive adds noarchive, which Google no longer uses and Bing reads as “keep this out of Copilot answers.” No Snippet adds nosnippet, which removes the text snippet from results and keeps the page’s text out of AI Overviews as a direct input. Leave the field empty to follow your site-wide defaults.
What does “index, follow, max-image-preview:large” mean?
It is the default robots tag Yoast and many other plugins print on public pages: index the page, follow its links, and allow large image previews. The full default usually adds max-snippet:-1 and max-video-preview:-1, meaning no limit on snippet or video preview length. All of these are valid, documented values, and together they are the recommended setting for any page you want found.
Is the noodp directive still useful?
No. noodp instructed engines not to pull titles and descriptions from the DMOZ Open Directory Project, which shut down in 2017. Google ignores it entirely now. If an inherited plugin or template still emits noodp or noydir, it’s harmless legacy clutter you can safely remove from your configuration.
How do I check which robots directive Google actually sees?
Use the URL Inspection tool in Google Search Console. It shows the rendered robots directive Google read, which can differ from your source if a tag is injected dynamically or inherited from a global rule. For X-Robots-Tag headers, run curl -I <url> since headers never appear in page source, and confirm the directive value matches your intent.
Meta robots advanced settings are where indexing strategy gets surgical, and where a single careless default can erase a site’s visibility. Treat every directive as a deliberate decision tied to a page type, verify the rendered output, and keep the access-versus-presentation distinction front of mind. For adjacent foundations, see SEO site structure.