# Set your AI crawler policy in robots.txt

> Decide which AI bots can read your site, write it in robots.txt, and make sure nothing else silently blocks them.

Canonical: https://abacklaunch.com/prompts/ai-crawler-access
Last updated: 2026-10-26

Version 1.0.

Whether the production site shows up in ChatGPT, Claude, Perplexity and Gemini answers depends partly on which AI crawlers can read it. Many sites block them by accident (a default firewall rule, a copied robots.txt) or allow everything without meaning to. Set a deliberate policy: AI model training allowed or blocked as I decide (ask me before writing it), AI search and user-requested fetches allowed, and make every layer agree with it.

Site: not given. Find the production URL in the project config, or ask me.

## Ground rules

1. Read before you write. Explore the parts of the codebase this job touches and tell me what is already in place.
2. Plan, then stop. Give me a numbered plan: what you will change, in which files, and why. Wait for my approval before editing anything.
3. Fix things at their source (the shared layout, the config, the generator), not with one-off patches page by page.
4. Keep the project's conventions: its framework APIs, file layout, naming and code style. Add no new dependency without asking.
5. Where something is already done well, say so and leave it alone.

## You will not

- Edit `.env` files, secrets or API keys, or print their values.
- Deploy, push, merge or publish anything.
- Delete data, or run migrations or scripts against a production database.
- Add tracking, third-party scripts or links to any site I did not ask for.
- Change copy, design or behaviour that this job does not need.
- Block Googlebot or Bingbot.
- Change firewall or CDN settings outside this repository.

## Your stack

- Framework: Work out the framework, router and build output from the project files (package.json, lockfiles, framework config) before planning, and say which you found.
- Host: Work out where the site is hosted from the config files (vercel.json, netlify.toml, wrangler config, Dockerfile, fly.toml) and say which you found.

## Steps

### 1. Read the current state

Show me the current robots.txt (or how it is generated), any `X-Robots-Tag` headers, meta robots tags, and anything that blocks bots before the app sees them: middleware, bot protection, a WAF or the host's 'block AI bots' setting if it is in the repository. Note anything I need to check in a dashboard.

### 2. Write the policy as named groups

Write robots.txt with explicit groups, so the intent survives a future edit to `User-agent: *`:

- **AI search and answers (allow):** `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`.
- **Fetches a person asked for (allow):** `ChatGPT-User`, `Claude-User`, `Perplexity-User`.
- **Model training (AI model training allowed or blocked as I decide (ask me before writing it)):** `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `meta-externalagent`.

Each group keeps the same `Disallow` lines as `*` for private paths (admin, account, API, checkout). Note in a comment that `Google-Extended` does not affect Google Search or AI Overviews, which use Googlebot.

### 3. Make the other layers agree

If middleware or bot protection blocks user agents, align it with the policy above. If the host or CDN has an 'AI bots' setting you cannot change from code, write down exactly what to set. Check that public pages do not send `noindex` or `X-Robots-Tag: noai` by mistake.

### 4. Say it in plain words too

Add an `/ai.txt` (or a section in an existing policy page) that states the same choices in plain language: what AI systems may do, what they may not, how to attribute, and a contact. Keep it in sync with robots.txt: add a comment in both files pointing at the other.

## Done means

- [ ] robots.txt has named groups for AI search, user fetches and training, with the training choice applied.
- [ ] Private paths are disallowed for every group.
- [ ] No middleware, firewall rule or header contradicts robots.txt.
- [ ] An ai.txt or policy section states the same choices in plain language.

## Report back

When you finish, give me:

- what you changed, file by file;
- anything you skipped, and why;
- anything I must do by hand (DNS records, dashboards, accounts, deploys);
- how to check each change on the live site.

## Related

- [Make your site readable by AI assistants: llms.txt and Markdown versions](https://abacklaunch.com/prompts/llms-txt-and-markdown)
- [Definition and FAQ blocks that answer engines can quote](https://abacklaunch.com/prompts/answer-ready-content)
- [Entity links: Organization, founder and official profiles](https://abacklaunch.com/prompts/entity-and-founder-links)