The X-Robots-Tag is an HTTP response header that tells search engines how to index and crawl a resource — and unlike the meta robots tag, it works on any file type, not just HTML. That makes it the only clean way to keep PDFs, images, JSON endpoints, and ZIP downloads out of the index. If you have ever watched a stray /exports/report.pdf rank for a branded query you would rather hide, the X-Robots-Tag is the fix.
X-Robots-Tag
The X-Robots-Tag is an HTTP response header that instructs search engine crawlers how to index and crawl a resource — supporting directives like noindex, nofollow, noarchive, nosnippet, and unavailable_after — and it works on both HTML and non-HTML files because it ships with the server response itself.
Why the X-Robots-Tag Exists
The meta robots tag lives inside an HTML <head>. That is fine until you need to control something that has no <head> — a PDF whitepaper, a generated CSV, a thumbnail, a webhook payload, a Server-Sent Events stream. None of those can carry a meta tag. The X-Robots-Tag solves this by moving the instruction up into the HTTP response, so it travels with the file regardless of format.
This is also why it is the heavier instrument. A meta tag is a content edit; an X-Robots-Tag is a server or edge config change that can blanket an entire directory or content type in one rule. Used well, that is leverage. Used carelessly, it is how a single regex noindexes your whole site. We have cleaned up both outcomes.
The directives you will actually reach for:
- noindex — keep the resource out of the index
- nofollow — do not follow links found in the resource
- none — shorthand for
noindex, nofollow - noarchive — suppress the cached copy
- nosnippet — suppress text, image, and video snippets
- unavailable_after — drop the resource from the index after a date
- max-snippet, max-image-preview, max-video-preview — cap how much Google shows
Combine them in one comma-separated header (X-Robots-Tag: noindex, noarchive) or split across multiple headers. You can also target a specific crawler by prefixing its user-agent token: X-Robots-Tag: googlebot: noindex.
One thing the header does not do: block crawling.
noindextells a crawler “do not list this,” but the crawler still has to fetch the file to read the header. If a page is blocked in robots.txt, Google never sees thenoindex— which is the classic reason a “noindexed” URL stubbornly stays in the index. Let it be crawled, then noindex it.
X-Robots-Tag vs. Meta Robots vs. robots.txt
These three controls get conflated constantly. They operate at different layers and answer different questions.
| Control | Layer | Works on non-HTML? | Stops crawling? | Stops indexing? |
|---|---|---|---|---|
| X-Robots-Tag | HTTP response header | Yes | No | Yes (noindex) |
| Meta robots | HTML <head> | No | No | Yes (noindex) |
| robots.txt | Site-root directive | Yes (by path) | Yes | No (URL can still index if linked) |
The practical rule: robots.txt manages crawl budget, the robots directives manage indexing. If you want a resource gone from search results, you need a noindex (via header or meta) that Google can actually reach — not a robots.txt Disallow. For the HTML-tag side of this and its noindex,follow nuances, see the meta robots advanced settings. For the full removal playbook, how to unindex pages from search engines walks the decision tree.
Setting the X-Robots-Tag by Platform
These are the configs we actually deploy. Test every one of them after rollout — a misplaced directive is the single most common self-inflicted indexing wound, and it shows up in Search Console as discovered, currently not indexed or worse.
Apache (.htaccess or vhost)
Block indexing of all PDFs site-wide using a files match:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, noarchive"
</FilesMatch>For a single file, scope it to that path:
<Files "secret.pdf">
Header set X-Robots-Tag "noindex, nofollow"
</Files>Nginx
Target a file type with a location block:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, noarchive";
}For one specific file:
location = /files/secret.pdf {
add_header X-Robots-Tag "noindex, nofollow";
}IIS (web.config)
Add a custom response header. To scope it to a file type, pair this with a URL Rewrite outbound rule rather than the global header below:
<configuration>
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="X-Robots-Tag" value="noindex, noarchive" />
</customHeaders>
</httpProtocol>
</system.webServer>
</configuration>PHP / application level
Send the header conditionally — ideal when the rule depends on auth state, a query parameter, or a feature flag:
header('X-Robots-Tag: noindex, noarchive');Cloudflare
Use a Transform Rule (Modify Response Header) or a Worker to inject the header per path or content type at the edge:
export default {
async fetch(request, env, ctx) {
const response = await fetch(request);
const newResponse = new Response(response.body, response);
if (new URL(request.url).pathname.endsWith('.pdf')) {
newResponse.headers.set('X-Robots-Tag', 'noindex, noarchive');
}
return newResponse;
}
};Netlify — set headers in _headers:
/files/*.pdf
X-Robots-Tag: noindex, noarchiveAWS S3 / CloudFront — attach a Response Headers Policy in the CloudFront behavior that adds X-Robots-Tag: noindex, noarchive, or set per-object metadata on upload (aws s3 cp --metadata-directive REPLACE --metadata x-robots-tag=noindex).
A note on the edge
When a CDN sits in front of your origin, the final response header wins. If your origin emits one X-Robots-Tag and your edge rewrites another, the browser and crawler see the edge value. Pick one layer and make it authoritative — we default to the edge for uniform behavior across a media-heavy crawl budget footprint, then strip any conflicting origin header.
When to Reach for It (and When Not To)
Use the X-Robots-Tag when:
- The asset is non-HTML — PDFs, images, video, XML, JSON, ZIP — and cannot carry a meta tag.
- The content is generated by a back-end script or API where editing the markup is impractical.
- You want one server rule to govern an entire directory or file type without touching each file.
- The decision is conditional — auth state, environment (staging vs. production), or a launch embargo with
unavailable_after. - Different crawlers need different rules, via user-agent-prefixed directives.
Reach for an alternative when:
- It is a standard HTML page you can edit. A meta robots tag is clearer for editors and lives next to the content. The header is overkill.
- You are trying to block access, not indexing. Use authentication or robots.txt — the header controls how a fetched resource is treated, not whether it can be fetched.
- The URL is blocked in robots.txt. Then your
noindexnever gets read. Unblock crawling first.
Validate every change. curl -I https://example.com/file.pdf shows the live header in one line; DevTools → Network confirms it in the browser; and Search Console’s URL Inspection (run on the canonical HTML that links the asset) confirms how Google actually treated it. Pair this work with clean XML sitemaps so you are only asking Google to index the URLs you want, and with a sound grasp of crawlability so the directives you set are reachable in the first place. If you want a second set of eyes on a large header migration, our technical SEO services cover exactly this.
Frequently Asked Questions
What is the difference between X-Robots-Tag and meta robots?
Both carry the same directives (noindex, nofollow, noarchive), but the meta robots tag only works inside HTML pages, while the X-Robots-Tag is an HTTP response header that works on any file type — PDFs, images, JSON, video. Use meta robots for editable HTML and X-Robots-Tag for non-HTML assets.
Does X-Robots-Tag block Google from crawling a page?
No. The X-Robots-Tag only controls how a resource is indexed and shown, not whether it is crawled. Google must still fetch the file to read the header. To stop crawling, use robots.txt — but note a robots.txt-blocked URL can never have its noindex read, so it may still appear in results.
How do I add an X-Robots-Tag noindex to a PDF?
Add a server rule that matches .pdf files and emits the header. In Apache, wrap a <FilesMatch "\.pdf$"> block with Header set X-Robots-Tag "noindex, noarchive"; in Nginx, use a location ~* \.pdf$ block with add_header. Then verify with curl -I against the file URL.
How do I check if an X-Robots-Tag header is working?
Run curl -I https://example.com/file.pdf and look for the X-Robots-Tag line in the response headers, or open DevTools → Network and inspect the response. For HTML, use Search Console’s URL Inspection tool, which reports the indexing directives Google read on its last crawl.
Can X-Robots-Tag target specific search engine crawlers?
Yes. Prefix the directive with the crawler’s user-agent token: X-Robots-Tag: googlebot: noindex applies only to Googlebot, while a separate X-Robots-Tag: bingbot: noindex, nofollow line can set different rules for Bing. Crawlers ignore directives prefixed with a user agent that does not match their own. Understanding how Googlebot and other web crawlers read these headers keeps the rules predictable.