robots.txt is a plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which paths they may fetch and which they should skip. It is made of groups: a User-agent line naming a crawler, followed by Allow and Disallow rules for paths, plus optional Sitemap lines. To see what your current file allows for Google, Bing and the AI crawlers on any page, paste the URL into the free robots.txt tester.
The file is small, but a single wrong line can hide a whole site from search. It also became a business decision in the last two years, because it is where you say yes or no to AI crawlers such as GPTBot and ClaudeBot. This guide covers the syntax with examples, a setup that fits most SaaS products, how to handle each AI crawler, the mistakes we see most often, and how to test the file before and after you ship it. If you would rather start from a form than a blank file, the free robots.txt generator builds one for you.
What robots.txt does (and what it does not)
Crawlers that follow the Robots Exclusion Protocol, standardized in 2022 as RFC 9309, request /robots.txt before they fetch anything else on a host. They read the group that applies to them and skip the paths it disallows. Our robots.txt glossary entry has the short version. Three limits matter more than the syntax:
- It controls crawling, not indexing. If other sites link to a blocked URL, Google can still index the bare address without reading the page. Search Console reports this as "Indexed, though blocked by robots.txt".
- It is a request, not a lock. Reputable crawlers honor it. Scrapers and bad bots ignore it. Anything private needs a login, not a Disallow line.
- It is public. Anyone can read your robots.txt, so listing
/secret-admin-panel/in it is an invitation.
If you want a page out of Google's results, use a noindex tag and leave the page crawlable, because Google has to fetch the page to see the tag. The noindex tag guide covers that side in detail.
robots.txt syntax, line by line
Here is a complete robots.txt example with every directive you are likely to need:
# Rules for every crawler that has no group of its own
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /admin/help/
# A stricter group for one crawler
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
User-agent
The User-agent line starts a group and names the crawler it applies to. * means every crawler. A crawler uses only the most specific group that names it and ignores the rest. That surprises people: if you write a User-agent: Googlebot group, Googlebot stops reading the * group entirely, so any rule you want it to follow must be repeated inside its own group. You can stack several User-agent lines on top of one set of rules to apply them to several crawlers at once.
Disallow and Allow
Disallow blocks a path prefix. Disallow: /admin/ blocks /admin/, /admin/users and everything under it. An empty Disallow: blocks nothing. Allow carves an exception out of a broader block. When two rules match the same URL, the longest (most specific) rule wins, and if they are the same length, Allow wins. Paths are case sensitive: /Admin/ and /admin/ are different rules.
Wildcards: * and $
Google, Bing and other major crawlers support two special characters in paths. * matches any run of characters, and $ anchors the end of the URL.
| Rule | Blocks | Does not block |
|---|---|---|
Disallow: /*?sort= | /pricing?sort=asc | /pricing |
Disallow: /*.pdf$ | /files/deck.pdf | /files/deck.pdf?v=2 |
Disallow: /app | /app, /app/, /apple | /blog/app |
Disallow: /app/ | /app/, /app/settings | /app, /apple |
The third row is a classic bug: a rule without a trailing slash is a prefix, so Disallow: /app also blocks /apple and /application. Add the slash when you mean a folder.
Sitemap
The Sitemap line gives the full URL of your XML sitemap. It sits outside any group, you can list several, and it must be an absolute URL. It is the easiest way to tell every crawler where your sitemap lives without submitting it to each search engine. The XML sitemap guide explains what should be in that file.
Directives Google ignores
- Crawl-delay: Google ignores it. Bing and some others respect it.
- Noindex in robots.txt: Google stopped supporting it in 2019. Use the meta tag or X-Robots-Tag header instead.
- Host: an old Yandex directive with no effect on Google.
Disallow all, allow all, and the lines in between
The two extremes are short, and they are easy to confuse.
# Disallow all: block every crawler from the whole site
User-agent: *
Disallow: /
# Allow all: an empty Disallow (or no robots.txt at all)
User-agent: *
Disallow:
The only difference is one slash. "Disallow all" is right for a staging site or a preview deployment, and it is the single most common reason a newly launched site gets no search traffic: the staging robots.txt ships to production. If you use disallow all on staging, add a check to your deploy so it never reaches the live domain. Better still, protect staging with a password, which works against every crawler and keeps it out of the index too.
A robots.txt for a typical SaaS site
Most SaaS products have the same shape: public marketing pages you want crawled, and an app, an API and account pages that crawlers have no reason to fetch. Blocking the second group saves crawl budget for the pages that can rank, and keeps login screens and empty dashboards out of reports.
User-agent: *
# The logged-in product and its API
Disallow: /app/
Disallow: /api/
Disallow: /dashboard/
Disallow: /settings/
# Auth flows and checkout
Disallow: /login
Disallow: /signup/verify
Disallow: /checkout/
# Internal search and filter combinations
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
# Public API docs live under /api/, so let them through
Allow: /api/docs/
Sitemap: https://example.com/sitemap.xml
A few notes on adapting this:
- Keep /login and /signup crawlable if they should rank. Many people search "yourproduct login". Blocking the login page can make that search worse, so only block the steps behind it.
- Subdomains need their own file. robots.txt applies per host. If your app runs on app.example.com, its rules go in app.example.com/robots.txt, and the file on example.com does nothing for it.
- Never block CSS, JavaScript or images your pages need. Google renders pages like a browser. If it cannot load your scripts, it may see a blank or broken page.
- Do not block pages you have set to noindex or canonicalized. Google must crawl them to see those signals. See the canonical tag guide for why.
robots.txt for AI crawlers
AI companies crawl the web with their own user-agents, and most of them now split crawling into separate jobs: collecting training data, building a search index, and fetching a page because a user asked. Each job has its own token, so you can block training while staying visible in AI answers. These are the tokens the companies document as of November 2026:
| Company | User-agent token | What it does |
|---|---|---|
| OpenAI | GPTBot | Collects content that may be used to train OpenAI's models |
| OpenAI | OAI-SearchBot | Builds the index that surfaces sites in ChatGPT search |
| OpenAI | ChatGPT-User | Visits a page when a ChatGPT user's request needs it; OpenAI says robots.txt may not apply |
| Anthropic | ClaudeBot | Collects content that may be used for model training |
| Anthropic | Claude-SearchBot | Indexes content to improve Claude's search results |
| Anthropic | Claude-User | Fetches pages when a Claude user asks a question |
| Google-Extended | A control token, not a separate crawler: says whether Google may use your content for Gemini training and grounding. It does not affect Google Search | |
| Perplexity | PerplexityBot | Indexes sites for Perplexity's answers and search results |
Two points are easy to miss. First, blocking Google-Extended does not remove you from AI Overviews or AI Mode, because those features are part of Google Search and use Googlebot. Second, blocking GPTBot does not remove you from ChatGPT search, because that is OAI-SearchBot's job. Names change, so check each company's crawler documentation before a big change.
Three common AI crawler policies
Allow everything (the default when you write nothing). Best for most startups: you want assistants to read, quote and link you. If that is your goal, also read how to get cited by ChatGPT and Perplexity.
Block training, allow search. A middle path many publishers take:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Block all AI crawlers. List every token above with Disallow: /. Know the cost: you will be cited less, or not at all, in AI answers, and user-triggered fetchers may still visit.
The robots.txt generator has a switch for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, so you can choose a policy without writing the groups by hand. To check the rest of your AI visibility (crawler access, llms.txt, schema and page structure), run the GEO checker.
Common robots.txt mistakes
- Shipping "Disallow: /" from staging. Covered above, and still the number one cause.
- Using robots.txt to remove pages from Google. A block stops Google from seeing your noindex, so the URL can stay indexed. Use noindex and leave it crawlable.
- Writing a specific group and forgetting it replaces the * group. A
User-agent: Googlebotgroup with one rule means Googlebot ignores every rule under*. - Missing trailing slashes.
Disallow: /blogalso blocks /blog-archive and /blogroll. - Wrong location or format. The file must be at the root of the host, named in lowercase, served as plain text with a 200 status. A robots.txt in a subfolder is ignored.
- Server errors on robots.txt. If robots.txt returns a 5xx error, Google treats the site as temporarily fully disallowed and slows or pauses crawling. A 404, on the other hand, means "no rules": everything is allowed.
- Relative sitemap URLs.
Sitemap: /sitemap.xmlis invalid. Use the full https:// address. - A huge file. Google reads only the first 500 KiB. If yours is that long, it is trying to do a job that belongs in your site's URL design.
How to test robots.txt
Google retired the old robots.txt Tester in Search Console in 2023 and replaced it with a robots.txt report, which shows the files Google found, when it last fetched them and any parse errors. It does not let you test a URL against a rule. For that, use a robots.txt tester.
Test with the free robots.txt tester: paste any page URL and it reads robots.txt from that site, then shows whether Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot may crawl that exact path, the rule that decides each verdict, and the sitemaps the file lists with how many URLs each holds. A good routine:
- Test your home page, a blog post and your pricing page. All should be allowed for Googlebot.
- Test one app or API URL. It should be blocked if you meant to block it.
- Check the AI crawler verdicts match the policy you chose.
- Confirm your sitemap is listed and has URLs in it. The sitemap checker goes deeper.
- Retest after every deploy that touches robots.txt, and keep the file in version control so you can see who changed what.
Google usually caches robots.txt for up to 24 hours, so a fix may take a day to be picked up. You can ask for a faster refetch from the robots.txt report in Search Console.
FAQ
Where do I put my robots.txt file?
At the root of each host, so it loads at https://yourdomain.com/robots.txt. Every subdomain (www, app, docs, blog) needs its own file, and http and https are treated as separate hosts too.
Does robots.txt stop a page from being indexed?
No. It stops crawling. A blocked URL can still appear in search results if other pages link to it. To keep a page out of Google, add a noindex meta tag or X-Robots-Tag header and let Google crawl the page so it can see the tag.
How do I disallow all crawlers?
Use User-agent: * followed by Disallow: /. This blocks every crawler that respects robots.txt from the whole site. Use it for staging, never for a live site you want found.
Should I block AI crawlers in robots.txt?
For most startups, no: being read and cited by ChatGPT, Claude and Perplexity sends you visitors. If you object to training, block the training tokens (GPTBot, ClaudeBot, Google-Extended) and keep the search ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed.
Is there a free robots.txt generator?
Yes. The free robots.txt generator on Aback Launch lets you pick folders to block, decide on GPTBot, ClaudeBot, PerplexityBot and Google-Extended, add your sitemap, and copy the finished file.
How long does it take Google to see robots.txt changes?
Google generally caches robots.txt for up to a day. After that, new rules apply to new crawls. Pages already indexed stay indexed until Google recrawls them or you remove them another way.
Ship it, then check it
A good robots.txt for a young product is short: block the app, the API and the filter noise, list your sitemap, and make a deliberate choice about AI crawlers. Build yours with the robots.txt generator, then check every important page with the free robots.txt tester before your launch goes live.
Topics
Written by
Devvrat Hans
Founder
By subscribing, you agree to get our newsletter by email. Unsubscribe any time in one click. See our Privacy policy and Terms.
Related Articles
301 Redirect Generator Guide: 301 vs 302, and Redirect Code for Apache, Nginx, Next.js and Cloudflare
How to use a 301 redirect generator, and when a 301 is the right redirect at all: 301 vs 302 vs 307 vs 308, copy and paste code for .htaccess, Nginx, Next.js and Cloudflare, and how to avoid chains and loops in a migration.
Bulk Domain Authority Checker: How to Compare Many Domains at Once (Free)
When to check domain authority in bulk, how to compare up to 20 domains at once for free, how to read and sort the results, and a link prospecting workflow with spreadsheet formulas.
Canonical Checker Guide: What a Canonical Tag Does and How to Check Yours
How to use a canonical checker, and what it is checking for: what rel canonical means, HTML and HTTP header examples, when to use a canonical tag, the mistakes that drop pages from Google and how to read Search Console's canonical reports.