Skip to content
Aback Launch
Guides11 min read

Noindex Tag: How to Keep a Page Out of Google (Meta Robots, X-Robots-Tag and robots.txt)

How the noindex tag works: the meta robots and X-Robots-Tag code, why robots.txt Disallow is not noindex, which pages to noindex, how to remove a page from Google fast, and how to check it.

Devvrat Hans

Founder

November 19, 2026
Noindex Tag: How to Keep a Page Out of Google (Meta Robots, X-Robots-Tag and robots.txt)

A noindex tag tells search engines not to show a page in their results. You add it as a meta robots tag in the page's head, <meta name="robots" content="noindex">, or as an X-Robots-Tag: noindex HTTP header, and Google drops the page the next time it crawls it. The page must stay crawlable for this to work: if robots.txt blocks it, Google never sees the noindex. To see whether any page carries one right now, paste it into our free on-page SEO checker, which reads both the tag and the header.

Noindex is one of the most useful and most dangerous lines in SEO. Used on the right pages, it keeps thin and private pages out of search and lets Google spend its attention on the pages that matter. Left on by mistake (a staging setting that ships to production is the classic case), it quietly removes your whole site. This guide covers the code, the robots.txt confusion, when to use it, and how to check it.

What the noindex tag does

A noindex directive is a request to search engines: crawl this page if you like, but do not put it in your index. Google, Bing and the other major engines honor it. When Googlebot next fetches the page and finds the directive, it removes the page from results, even if other sites link to it.

Three things noindex does not do:

  • It does not stop crawling. Google still fetches the page to read the directive. Over time it tends to crawl a noindexed page less often.
  • It does not stop links from being followed, unless you also add nofollow. The default is "noindex, follow": hide this page, but use its links to find others. Google has said that a page left on noindex for a long time may eventually be treated as if its links were nofollow too, so do not rely on a noindexed page as the only route to important pages.
  • It does not protect private content. Anyone with the URL can still open the page. Anything confidential belongs behind a login.

To understand where noindex sits, it helps to separate two steps: crawling (fetching the page) and indexing (storing it so it can appear in results). robots.txt controls the first. Noindex controls the second.

Meta robots noindex: the HTML tag

For an HTML page, the simplest way to noindex it is a meta robots tag inside the <head>:

<meta name="robots" content="noindex">

That applies to every search engine. To target only Google, use the googlebot name instead of robots:

<meta name="googlebot" content="noindex">

You can combine directives with commas. The ones you are likely to need:

DirectiveWhat it doesWhen to use it
noindexKeep the page out of resultsThin, duplicate, private or utility pages
nofollowDo not follow the links on this pageRarely; pages full of links you do not vouch for
noindex, followHide the page, still follow its linksThe usual choice; same as noindex alone
noneSame as noindex, nofollowPages you want ignored entirely
nosnippetShow the page, but no text snippetPages where the snippet would mislead
unavailable_after: [date]Drop the page from results after a dateTime-limited offers, event pages

Google's full list is in its robots meta tag documentation. "index" and "follow" are the defaults, so you never need to write content="index, follow". It does no harm, but it does nothing.

Setting noindex in common frameworks

Most startup sites do not write head tags by hand. In Next.js (App Router), set it in the page's metadata:

export const metadata = {
  robots: { index: false, follow: true },
};

In WordPress, the reading setting "Discourage search engines from indexing this site" adds a noindex to every page, and SEO plugins such as Yoast and Rank Math have a per-page toggle. In Webflow and Framer, look for a per-page "exclude from search" or SEO setting. Whatever the platform, check the output, not the setting: view the page source and search for "noindex".

Do not add noindex with JavaScript only

If the HTML your server sends contains a noindex, Google may skip rendering the page at all, so a script that later removes the tag may never run for Googlebot. The reverse (adding noindex with JavaScript) is unreliable too. Put the directive, or its absence, in the server HTML or the HTTP header.

X-Robots-Tag: noindex in the HTTP header

The X-Robots-Tag is the same directive sent as an HTTP response header instead of a tag. It accepts the same values:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

Use the header when you cannot add a meta tag, which is any file that is not HTML: PDFs, images, CSV exports, API responses. It is also handy for noindexing a whole folder or host from the server config. In Nginx:

location /internal-docs/ {
  add_header X-Robots-Tag "noindex" always;
}

In Apache, inside .htaccess or the site config:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

The header is easy to forget because you cannot see it in the page source. Hosting platforms and CDNs sometimes add it to preview deployments, which is correct there and a disaster if the same rule reaches your real domain. To see the headers a URL actually returns, run it through the free HTTP header checker, which lists every response header, X-Robots-Tag included.

Robots.txt no index: why Disallow is not noindex

This is the most common noindex mistake, and the reason "robots.txt no index" gets searched so often. robots.txt can stop crawling. It cannot stop indexing.

Google used to tolerate an unofficial Noindex: line in robots.txt, but it stopped supporting it on September 1, 2019. Today a line like this does nothing in Google:

User-agent: *
Noindex: /thank-you/

And this does something you probably do not want:

User-agent: *
Disallow: /thank-you/

Disallow tells Googlebot not to fetch the page. If the page is already indexed, or other pages link to it, Google can keep it in results, shown with no description because Google is not allowed to read it. In Search Console that shows up as "Indexed, though blocked by robots.txt". Worse, if the page also has a noindex tag, Google never sees the tag, because it is not allowed to fetch the page.

Noindex vs disallow at a glance

Noindex (meta or header)robots.txt Disallow
Stops crawlingNoYes
Keeps the page out of resultsYesNo, a linked URL can still appear
Works for files that are not HTMLYes, with X-Robots-TagYes
Saves crawl budgetBarelyYes
Use it forPages that must not appear in searchLarge areas not worth crawling at all

The rule: to remove a page from results, allow crawling and add noindex. Use robots.txt Disallow only for things you do not want fetched at all, such as endless filter combinations or internal search results on a big site, where saving crawl budget matters. Never use both on the same URL while you are trying to get it removed. Our free robots.txt tester shows whether Googlebot is allowed to fetch a given URL and which rule decides, so you can confirm a page you are noindexing is not also blocked.

When to noindex a page

A small site rarely needs many noindexed pages. Noindex the pages that would disappoint a searcher who landed on them, and leave everything else alone.

Good candidates

  • Internal search results. Pages like /search?q=crm are thin, endless and repeat your real pages. Google has long advised against letting internal search result pages show up in its results.
  • Thank-you and confirmation pages. Nobody searches for "thanks for signing up", and an indexed one can inflate conversion tracking with visitors who never converted.
  • Staging and preview environments. Noindex is the minimum; password protection is better, because it also keeps people and other crawlers out.
  • Thin tag and archive pages. A blog tag page with one post on it adds nothing. Either noindex it or give it real content.
  • Account, cart and checkout pages. Usually behind a login already, but the public parts (login, password reset) do not need to rank.
  • Paid campaign landing pages that duplicate your main page with small changes for ads.
  • Old pages you keep for existing users but do not want new visitors to find, such as a legacy changelog or a retired feature page.

Pages you should not noindex

  • Duplicates of an indexed page. Use a canonical URL instead, so the signals consolidate on one version. Our canonical tag guide covers the difference. Do not put noindex and a canonical pointing elsewhere on the same page: they send mixed signals.
  • Pages you are moving. A page that has moved should 301 redirect to its new address, which passes its signals on. Noindex throws them away.
  • Paginated lists. Page 2 and beyond of a list are often the only route to older items. Let them be indexed and give each a self-referencing canonical.
  • Privacy policy and terms. They rarely rank for anything, but they do no harm and some people search for them by name.

How to remove a page from Google fast

Noindex works on Google's schedule: the page drops out the next time it is crawled, which can take days for a busy page or weeks for an obscure one. If you need it gone sooner, use these steps together:

  1. Add the noindex (or delete the page so it returns 404 or 410 if it should not exist at all).
  2. Make sure the URL is not blocked in robots.txt, so Google can fetch it and see the change.
  3. Hide it now with the Removals tool in Google Search Console. A temporary removal hides the URL from Google results for about six months. It does not remove anything from the index by itself, which is why step 1 has to be in place before the six months run out.
  4. Ask for a recrawl with URL Inspection and "Request indexing", so Google fetches the page and reads the noindex.
  5. Remove the URL from your sitemap. A sitemap is a list of pages you want indexed; a noindexed URL in it is a contradiction Search Console will flag. Our XML sitemap guide covers what belongs in one.

For Bing, the equivalent is the Block URLs tool in Bing Webmaster Tools. If something sensitive is exposed, take the page down or put it behind a login first; delisting is the second step, not the fix.

How to check a noindex tag

Check for noindex in three places, because it can hide in any of them.

  1. The page HTML. The SEO checker fails a page that says noindex in its meta robots tag or its X-Robots-Tag header, and passes it as "Indexable" otherwise. The meta checker shows the robots tag next to the title, description and canonical.
  2. The HTTP headers. The HTTP header checker shows the raw X-Robots-Tag, which matters for PDFs and for headers your host or CDN adds without telling you.
  3. robots.txt. The robots.txt tester confirms the page is crawlable, so the noindex can be read.

Then check Google's view. In Search Console, the Page indexing report lists every URL Google found with "Excluded by 'noindex' tag". Go through that list after every redesign or migration. If a page you want to rank is on it, remove the directive and request indexing. Our guide to getting a website indexed by Google walks through the rest of that report.

From the command line

If you prefer a terminal, these two commands show the header and the tag:

curl -sI https://example.com/page | grep -i x-robots-tag
curl -s https://example.com/page | grep -io '<meta[^>]*robots[^>]*>'

Common noindex mistakes

  • Shipping staging settings to production. A site launches with "discourage search engines" still ticked, or an environment variable that sets noindex everywhere. Add a check for noindex on your home page to your deploy process.
  • Blocking and noindexing the same URL. Covered above: the block hides the tag.
  • A noindex in the template. One line in a shared layout noindexes every page that uses it. Spot-check a page of each type (home, blog post, docs page, product page) after template changes.
  • Noindexing pages that pass your internal links. If your only links to key pages are on noindexed pages, Google may find them late. Link to important pages from indexed ones too.
  • Expecting instant results. Noindex takes effect on the next crawl, and adding it back to a page Google dropped takes a crawl too.

FAQ

What is a noindex tag?

A noindex tag is a meta robots tag, <meta name="robots" content="noindex">, placed in a page's head to ask search engines not to show that page in their results. The same directive can be sent as an X-Robots-Tag HTTP header, which also works for PDFs and other files.

Can I noindex a page with robots.txt?

No. Google stopped supporting a Noindex line in robots.txt in September 2019. A Disallow rule stops crawling, not indexing, and it also stops Google from seeing a noindex tag on the page. Allow crawling and use the meta tag or the header instead.

What is the difference between noindex and nofollow?

Noindex keeps the page itself out of search results. Nofollow asks search engines not to follow the links on the page. They are independent: most pages that need hiding should be "noindex, follow", so their links still help crawlers find your other pages.

How long does noindex take to work?

The page is dropped the next time Google crawls it, which can be a few days for a popular page or several weeks for a rarely visited one. Request indexing in Search Console to speed up the recrawl, and use the Removals tool if it has to disappear today.

Does noindex hurt the rest of my site's SEO?

No. Noindexing thin or utility pages does not harm your other pages and can help by keeping weak pages out of results. The risk is only a noindex on pages you want to rank, so check your important pages after every release.

Should I use the X-Robots-Tag or the meta robots tag?

Either works for HTML pages; use whichever your stack makes easier to manage. Use the X-Robots-Tag for anything that is not HTML (PDFs, images, feeds) and for whole folders set in the server config. Do not set conflicting values in both.

Check your pages now

Paste your home page and your most important landing page into the free on-page SEO checker: it tells you in seconds whether either one is noindexed by tag or header, along with every other on-page issue it finds. Then run the same URLs through the robots.txt tester to be sure Google can reach them. Two minutes of checking now is cheaper than finding out from a traffic graph.

Written by

Devvrat Hans

Founder

Share
Newsletter
Get the Monday launch email
The week's new launches and winners, in one short email.

By subscribing, you agree to get our newsletter by email. Unsubscribe any time in one click. See our Privacy policy and Terms.

Launch your startup on Aback Launch

Get a launch page that search engines and AI assistants can read, a place in the weekly launch list, and a review by a real person. Free to join.