Everything that keeps a page out of Google
Google can only index a URL it may crawl, that answers with a success status, and that does not ask to stay out. The checker reads each of those the way Googlebot would. It follows redirects, because Google indexes the URL a redirect ends on, not the one you entered. It reads robots.txt for the Googlebot group, the robots and googlebot meta tags, and the X-Robots-Tag header, which also works for PDFs and other files.
A canonical tag that points to another URL does not block indexing, but it tells Google which version you prefer. Google will usually index that one instead, so the checker flags it.
Blocked and noindex are not the same
Blocked means Googlebot may not crawl the URL: robots.txt disallows it, robots.txt itself answers with a server error, or the page returns an error status. A URL blocked by robots.txt can still appear in results with no description if other sites link to it.
Noindex means Google may crawl the page but is asked not to index it. Google has to crawl the page to see that rule, so never block a noindex page in robots.txt. If you want a page gone from Google, allow crawling and keep the noindex until it drops out.
What the checker cannot see
It reads the HTML your server sends. If JavaScript adds or removes a noindex tag after the page loads, Google may see something different, and Google can skip rendering a page whose HTML already says noindex. For the final word on one URL, use the URL Inspection tool in Search Console.
Questions
Why is my page not indexed by Google?
Common reasons are a noindex robots meta tag, an X-Robots-Tag header with noindex, a robots.txt rule that blocks Googlebot, a canonical that points to another URL, or an error status. This checker reads all of them at once. If none applies, Google may simply not have crawled the page yet.
Does robots.txt Disallow stop a page from being indexed?
No. It stops crawling. The URL can still be indexed from links, usually without a description. To keep a page out of results, use noindex and let Google crawl it.
What does the X-Robots-Tag header do?
It carries the same rules as the robots meta tag, in the HTTP response. It is the only way to noindex a PDF, an image or another file that has no HTML head.
Does noindex in a googlebot meta tag count?
Yes. Google obeys both name="robots" and name="googlebot". When they conflict, Google applies the more restrictive rule.
My robots.txt returns an error. Does that matter?
Yes. Google reads a server error or a 429 on robots.txt as "do not crawl anything" and stops crawling the site until it can read the file. A 404 is fine: it means there are no rules.
Which Googlebot rules does it test?
The group for Googlebot if your robots.txt has one, otherwise the group for all crawlers, with the longest matching rule winning, as Google applies them.
Guides for this
- Noindex Tag: How to Keep a Page Out of Google (Meta Robots, X-Robots-Tag and robots.txt)How the noindex tag works: the meta robots and X-Robots-Tag code, why robots.txt Disallow is not noindex, which pages to noindex, how to remove a page from Google fast, and how to check it.
- How to Get Your Website Indexed by Google FastHow to get a new website indexed by Google fast: set up Search Console, submit a clean sitemap, request indexing, earn the first links, and fix the problems that keep pages out.
- robots.txt Guide: Syntax, Examples, AI Crawlers and the Mistakes to AvoidWhat robots.txt does, how to write one line by line, a ready robots.txt example for a SaaS site, how to handle GPTBot, ClaudeBot, Google-Extended and PerplexityBot, and the mistakes that quietly block Google.
Ask AI about this page
Open your assistant with a prompt to summarize this page, already filled in.