A robots.txt Disallow is not noindex
robots.txt controls crawling. A Disallow line stops a crawler from fetching a page, but the URL can still appear in results if other pages link to it, usually with no description. To keep a page out of results, use noindex.
A crawler has to fetch the page to read the noindex rule. If robots.txt blocks the page, the rule is never seen and the page can stay indexed. Remove the Disallow line until the page has dropped out.
Meta tag or X-Robots-Tag header
The meta tag goes in the head of an HTML page. The X-Robots-Tag HTTP header does the same job for any file, including PDFs, images and videos, and your server can send it for a whole folder. Both accept the same rules.
Rules for one crawler
Use name="robots" for all crawlers, or a crawler's name, like googlebot, to target one. Without any tag, the default is index, follow. When rules conflict, Google applies the most restrictive one.
Questions
Does a robots.txt Disallow remove a page from Google?
No. It stops crawling, not indexing, so a blocked URL can still show in results. Use noindex, and let crawlers fetch the page so they see it.
Should I add index, follow?
There is no need. Index and follow are the defaults, so a page with no robots tag is indexed and its links are followed.
How long until a noindexed page drops out?
It drops out after Google crawls the page again and sees the rule. To speed that up, request indexing with the URL Inspection tool in Search Console.
What does max-snippet:-1 mean?
No limit: Google chooses the snippet length. max-snippet:0 means no text snippet at all, the same as nosnippet.
Guides for this
- Noindex Tag: How to Keep a Page Out of Google (Meta Robots, X-Robots-Tag and robots.txt)How the noindex tag works: the meta robots and X-Robots-Tag code, why robots.txt Disallow is not noindex, which pages to noindex, how to remove a page from Google fast, and how to check it.
- robots.txt Guide: Syntax, Examples, AI Crawlers and the Mistakes to AvoidWhat robots.txt does, how to write one line by line, a ready robots.txt example for a SaaS site, how to handle GPTBot, ClaudeBot, Google-Extended and PerplexityBot, and the mistakes that quietly block Google.
- Breakdown: Dofollow Backlink Benefits for New StartupsA practical breakdown of how dofollow backlinks for new startup launch pages can improve discoverability, authority signals, and qualified acquisition when combined with strong launch execution.