How it differs from robots.txt and llms.txt
robots.txt is a machine protocol for crawling. llms.txt is a reading guide in Markdown. ai.txt is a human readable policy: whether content may be used for training, for answering questions, whether citation is expected, and who to contact. There is no single standard yet, so treat it as a clear statement of intent rather than an enforced rule.
What to include
The site's name and a one line description. What is allowed: crawling, quoting, summarising, training. What you ask in return, such as a link to the source page when you are quoted. Pointers to robots.txt, llms.txt and your sitemap. A contact address. Keep it factual and avoid instructions designed to bias an assistant's answers.
Is it worth publishing?
It takes minutes, it documents your position on AI use in one place, and some agents look for it. Combine it with allowing AI crawlers in robots.txt if your goal is to be cited.
ai.txt checklist
Serve it as text/plain at the root, not behind a redirect or a login. Keep it short enough to read in a minute. Make the permissions match your robots.txt: an ai.txt that welcomes AI assistants while robots.txt blocks GPTBot and ClaudeBot sends a mixed message, and the crawler will follow robots.txt.
Date it and review it when your policy changes, for example when you start offering an API or decide to opt out of training. Link it from your llms.txt so agents that find one can find the other.
Example
# ai.txt for acme.com Allowed: crawling, quoting, summarising Citation: link to the page you quote Contact: hello@acme.com
Questions people ask
- Do I need ai.txt if I have llms.txt?
- No, but they do different jobs. llms.txt guides reading. ai.txt states usage terms.
Related terms
- llms.txtllms.txt is a Markdown file at the root of a website that gives large language models a short summary of the site and links to its most useful pages, so they can read it without parsing every page.
- robots.txtrobots.txt is a plain text file at the root of a site that tells crawlers which paths they may and may not fetch. It controls crawling, not indexing.
- Generative Engine Optimization (GEO)Generative Engine Optimization (GEO) is the practice of making your content easy for AI assistants such as ChatGPT, Claude, Perplexity and Google's AI Overviews to find, understand and cite in their answers.