To unindex pages from search engines, you serve a crawlable noindex signal (meta tag or X-Robots-Tag header), or you delete the URL and return a 410 or 404, then you wait for a recrawl. Google and Bing both honor these signals. The single most common mistake is blocking a URL in robots.txt and expecting it to drop: that prevents crawling, which means the search engine never sees the noindex and the URL can stay indexed indefinitely.
How to Unindex Pages From Search Engines
Unindexing is the deliberate removal of a URL from a search engine’s index using a crawlable noindex directive, an HTTP X-Robots-Tag header, or a 410/404 status, so the page no longer appears in organic results.
This is one of those tasks that looks like a one-liner and quietly burns a quarter of someone’s traffic when done wrong. We see it constantly on programmatic and faceted sites: thousands of low-value URLs blocked in robots.txt, all of them still sitting in the index with a “no information available” snippet. The fix is mechanical once you understand which signal does what.
Noindex vs robots.txt: the distinction that breaks everyone
These two do different jobs, and conflating them is the root cause of most failed deindexing.
noindextells search engines “you may crawl this, but do not index it.” It removes the URL from results. It requires the page to be crawlable.- robots.txt
Disallowtells crawlers “do not fetch this URL at all.” It controls crawling only. A disallowed URL can still be indexed (URL-only, no snippet) if other pages link to it, which is what Search Console reports as “Indexed, though blocked by robots.txt.”
The trap: if you Disallow a URL and put noindex on it, the crawler is blocked from ever fetching the page, so it never reads the noindex. The page stays indexed. To deindex with noindex, the URL must be open in robots.txt so the crawler can re-read it. A noindex: line inside robots.txt does nothing either: Google stopped supporting that unofficial rule in September 2019.
Rule of thumb: robots.txt saves crawling. It is the wrong tool for removal. If a URL is already indexed, robots.txt alone will not get it out, and may actively prevent the signal that would.
The methods, ranked by what they actually do
| Method | Stops crawling? | Removes from index? | Best for |
|---|---|---|---|
noindex meta tag | No | Yes (after recrawl) | HTML pages you keep but hide |
X-Robots-Tag header | No | Yes (after recrawl) | PDFs, images, feeds, non-HTML, bulk |
| 410 Gone | Drops on recrawl | Yes | Permanently deleted pages |
| 404 Not Found | Drops on recrawl | Yes | Deleted pages |
robots.txt Disallow | Yes | No (can stay URL-only) | Saving crawl budget on URLs nobody links to |
| Canonical tag | No | Soft consolidation only | Duplicate/near-duplicate variants |
| Auth / password | Yes | Yes (no access) | Staging, private, members-only |
| Google Search Console Removals | No | Temporary hide, about six months | Emergencies, while a real fix lands |
| Bing Webmaster Tools Block URLs | No | Temporary hide, up to 90 days | The same emergencies, on Bing |
The code to unindex a page in HTML (meta noindex)
Add this to the <head> of any page you want gone:
<meta name="robots" content="noindex">
follow is the default, so noindex alone and noindex, follow do the same thing; spelling follow out is harmless. The page must not be blocked in robots.txt, or the crawler can’t read the tag. This is the cleanest option when you’re keeping the page live for users but want it out of search. To target one engine only, swap robots for googlebot or bingbot.
X-Robots-Tag HTTP header (non-HTML and bulk)
For files where there’s no <head> to edit (PDFs, images, JSON, feeds, generated downloads), set the directive at the server level:
# nginx: noindex all PDFs
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}
# Apache: noindex a single file
<Files "private-report.pdf">
Header set X-Robots-Tag "noindex"
</Files>
This also scales for deindexing entire patterns of URLs without touching templates. It’s the same directive family as the meta tag; see meta robots advanced settings for the full syntax.
Delete and return 410 (or 404)
If a page is genuinely gone, the honest signal is 410 Gone. Google’s documentation treats every 4xx status except 429 the same way for indexing, so a 404 removes the URL too; in practice a 410 can drop marginally sooner, and the difference is small. Either way, do not redirect a deleted URL to your homepage; that’s a soft 404 waiting to happen, and Google may ignore it. Read more on handling Not found (404) cleanly.
410 or noindex? Use a 410 when the page should stop existing for everyone. Use noindex when people still need the page (a login, a thank-you page, a filtered view) and only search should lose it.
Canonical tags: consolidation only
A rel="canonical" is a hint. It’s the right tool for near-duplicate variants (print versions, tracking parameters, sort orders) where you want signals consolidated onto the primary URL, and it won’t reliably pull a page out of the index on its own. Pick one signal per URL: a canonical when the variant should hand its signals to the primary page, noindex when it should simply drop out. Combining noindex with a canonical that points elsewhere sends mixed messages. This is also how Google’s “duplicate without user-selected canonical” status gets resolved.
Authentication and the removal tools
Password / HTTP auth is the only method that guarantees a crawler never sees content; use it for staging and private resources, and make sure those URLs aren’t sitting in a sitemap or linked publicly.
Noindex vs Removals in Search Console: the Removals tool hides a URL from Google for about six months; noindex (or a 410) removes it for good once Google recrawls. Treat Removals as an emergency tourniquet (leaked personal data, an accidental publish) while the real fix propagates. If the page still returns 200 with no noindex when the six months end, it can come back.
How to unindex pages from Bing
Bing reads the same noindex meta tag, X-Robots-Tag header and 410/404 statuses, so the fix you ship for Google covers Bing too. Two Bing specifics:
- Block URLs in Bing Webmaster Tools hides a URL or directory from Bing’s results for up to 90 days. Bing’s own help text says that if the URL still returns 200 OK after those 90 days, it reappears.
- IndexNow lets you tell Bing (and the other participating engines) about removed URLs as well as new ones, so a 410 gets noticed without waiting for the next crawl. Google does not use IndexNow. Our Bing Webmaster Tools and IndexNow guide covers the setup.
How to noindex URLs in WordPress
Most WordPress sites do this through an SEO plugin, and WordPress itself already noindexes a few URLs by default.
- One post or page: in Yoast SEO, open the Advanced tab of the Yoast box and set “Allow search engines to show this page in search results?” to No. In Rank Math, open Advanced and tick No Index under Robots Meta. AIOSEO has the same switch.
- A whole content type or taxonomy (all tags, author archives, attachment pages): set it once in the plugin’s content type or taxonomy settings.
- The whole site: Settings → Reading → “Discourage search engines from indexing this site.” That is what “enable noindex” means in WordPress: since WordPress 5.3 the checkbox adds a
noindex, nofollowrobots meta tag to every page. Staging sites need it; a live site with it ticked disappears from search.
What WordPress noindexes on its own
WordPress core adds noindex, noarchive to wp-login.php and noindex to internal search results (/?s=), embeds and comment-reply URLs. If a crawler report from Moz, Semrush or Screaming Frog lists wp-login.php as a noindexed page, that is core doing its job: leave it, and leave wp-login.php crawlable so the tag can be read. WordPress’s default robots.txt blocks only /wp-admin/.
Noindex feed URLs in WordPress
Feeds are XML, so a meta tag has nowhere to go; send the header instead. Add this to a code snippets plugin or your child theme’s functions.php:
add_action( 'template_redirect', function () {
if ( is_feed() ) {
header( 'X-Robots-Tag: noindex, follow', true );
}
} );
The same X-Robots-Tag approach covers PDFs and other uploads, set in your server config as shown above.
The sequence we actually run
When we deindex pages at scale, the order matters more than any single tactic:
- If it’s sensitive or leaked, restrict access immediately (auth) and file a removal request in Search Console and Bing Webmaster Tools to buy time.
- For pages you’re keeping, add a crawlable
noindex(meta orX-Robots-Tag) and confirm the URL is not blocked in robots.txt. - For pages you’re deleting, return 410 (or 404) at the server.
- Pull the URLs out of your XML sitemap and remove internal links pointing to them; a clean sitemap speeds re-evaluation. See XML sitemaps and SEO.
- Check what you shipped. Paste a URL into our free SERP & AI preview: it reads the live HTML and flags a meta robots
noindex. It does not read HTTP headers, so checkX-Robots-Tagwithcurl -I <url>, and use URL Inspection in Search Console to see the directive Google itself read. - Wait for a recrawl. Nothing happens until the crawler revisits. You can check when Google last crawled a page to gauge timing.
- Leave robots.txt alone unless you need the crawl savings. A
Disallowadded after removal saves crawl budget, and it also stops the crawler from rereading thenoindex, so a blocked URL with outside links can return as “Indexed, though blocked by robots.txt.” Keepnoindexalone on anything other sites link to. - Monitor the Pages report in Search Console and your server logs; if URLs reappear, the signal was misconfigured, so recheck step 2.
Why deindexing matters in the AI Overviews era
A page that is out of the index can’t rank in classic results, and it can’t be cited in AI Overviews or AI Mode either. That cuts both ways. A bloated index full of thin, duplicate or stale URLs gives both ranking and AI answers more low-quality surface area to misrepresent your brand, so trimming junk is quality control for what the engines can say about you. A thin page that could answer a real question is usually worth strengthening before you hide it; noindex is for pages that should never rank.
Privacy note: your own console only controls your own site. If personal data was copied to someone else’s site,
noindexon yours does nothing to that copy; Google and Bing each take personal-information and legal removal requests through separate forms.
Common ways this goes wrong
- Blocking in robots.txt to deindex. The number-one error. The crawler can’t read your
noindex, so the URL lingers. Open it and let it drop. - Expecting instant results. Deindexing is recrawl-gated. Low-priority URLs can take weeks. The answer is patience or a removal request.
- Redirecting deleted pages to the homepage. Reads as a soft 404 and may keep the old URL alive.
- Leaving deindexed URLs in the sitemap. You’re telling Google “index this” and “don’t index this” at the same time. Pick one.
- Trusting canonical to force removal. It’s a hint. For unwanted thin pages, use
noindexand tighten the underlying thin content. - Shipping the “Discourage search engines” checkbox from staging to production. The whole site goes
noindex, nofollowovernight.
When unindexing becomes a business problem
Removing a handful of pages is an afternoon’s work with the sequence above.
It becomes worth paying for when the index no longer matches the site you meant to publish: Search Console lists far more indexed URLs than your sitemap holds, “Indexed, though blocked by robots.txt” runs into the thousands, a faceted or programmatic template keeps minting URLs, or a migration is coming and nobody knows which pages earn their place. The hard part is the call on each URL; the tags are easy.
On a fractional SEO engagement, the two founders pull clicks and backlinks per URL, decide what gets kept, rebuilt, noindexed, redirected or returned as a 410, ship the signals and watch both engines afterward. We pruned the Prawomat rebuild that way: 2,646 URLs down to about 1,370, each judged on its own clicks and backlinks, as one of three moves shipped in the same release.
Frequently Asked Questions
Should I use noindex or robots.txt to remove a page from Google?
Use noindex to remove an already-indexed page; it must stay crawlable so Google can read the directive. Use robots.txt Disallow only to prevent crawling and save crawl budget. Blocking in robots.txt stops Google from seeing your noindex, so the URL can stay indexed.
Should I use 410 instead of noindex?
Use a 410 when the page is gone for everyone and has no reason to exist. Use noindex when visitors still need the page and only search results should lose it. Both remove the URL from Google and Bing after a recrawl.
What is the difference between noindex and the Removals tool in Search Console?
The Removals tool hides a URL from Google’s results for about six months and changes nothing on your site. noindex is a permanent instruction on the page itself that takes effect once Google recrawls. Use Removals for speed in an emergency, and ship noindex or a 410 so the URL stays gone.
How long does it take to unindex a page?
It depends on recrawl frequency. High-priority URLs may drop within days; low-value or rarely crawled pages can take several weeks. Nothing happens until the crawler revisits the page and reads the new signal. To force urgency, submit a temporary removal in Google Search Console (or Block URLs in Bing Webmaster Tools) while the permanent noindex or 410 propagates.
Does robots.txt prevent a page from appearing in Google?
Not reliably. robots.txt blocks crawling and leaves indexing alone. A disallowed URL can still appear as a bare, snippet-less listing if other pages link to it, because Google indexes the URL without fetching its content. To keep a page out of results entirely, use a crawlable noindex directive or password-protect it.
Should wp-login.php be noindexed?
It already is. WordPress core adds noindex, noarchive to the login page, so an SEO tool that reports it as noindexed is reporting the default. Leave the tag in place and leave the page crawlable so search engines can read it.
How do I noindex pages in WordPress?
Use an SEO plugin (Yoast, Rank Math or AIOSEO) to set noindex per post, per page or per content type; it adds the robots meta tag for you. For feeds and files, send an X-Robots-Tag header. Confirm the URL isn’t also disallowed in robots.txt.