How it works
Before crawling a site, a well behaved bot fetches /robots.txt and reads the group of rules for its user agent. Each group lists Disallow and Allow paths. The most specific matching rule wins. A Sitemap line anywhere in the file points crawlers to your sitemap.
The file is public and voluntary. It keeps polite bots out of areas that waste their time, but it is not security. Anything secret needs authentication, not a Disallow line.
AI crawlers
AI companies run their own crawlers, such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. You can allow or block each by name. Blocking them keeps your content out of some AI training sets and answers. Allowing them gives AI assistants a chance to read and cite you, which is the basis of AI search visibility.
Common mistakes
Disallow: / left over from staging, which blocks the whole site. Blocking CSS and JavaScript, so Google cannot render pages. Blocking a page to remove it from results, which stops Google from seeing its noindex. Typos in paths, which are case sensitive.
Testing your file
After every change, test the paths that matter: your homepage, a product page, a blog post, your sitemap and an asset such as a stylesheet. Check them for Googlebot and for the AI crawlers you care about. A robots.txt tester shows which rule matched each path, which catches surprises such as a wildcard that blocks more than you meant.
Remember that each subdomain has its own robots.txt. Rules on www.example.com do not apply to app.example.com or blog.example.com.
Example
User-agent: * Disallow: /dashboard/ Sitemap: https://abacklaunch.com/sitemap.xml
Questions people ask
- Does robots.txt remove a page from Google?
- No. A blocked page can still be indexed from links pointing to it. Use noindex and let the page be crawled to remove it.
Related terms
- NoindexNoindex is a directive that asks search engines to keep a page out of their results. It is set with <meta name="robots" content="noindex"> or an X-Robots-Tag HTTP header.
- XML sitemapAn XML sitemap is a file that lists the URLs you want search engines to crawl and index, optionally with the date each page last changed.
- Crawl budgetCrawl budget is the number of URLs a search engine is willing and able to fetch from your site in a given time. It matters most for large sites and sites with many low value URLs.
- llms.txtllms.txt is a Markdown file at the root of a website that gives large language models a short summary of the site and links to its most useful pages, so they can read it without parsing every page.
- ai.txtai.txt is a plain text file at a site's root that states, in plain language, how AI systems may use the site's content and which facts they should know about it.