Make your site readable by AI assistants: llms.txt and Markdown versions

Paste into any coding agent: Claude Code, Cursor, Codex, Copilot, Windsurf or Cline.

# Make your site readable by AI assistants: llms.txt and Markdown versions

AI assistants and agents read the production site to answer questions about it. Raw HTML full of navigation, scripts and styling is a poor source for them. Add the files and versions they look for: an `/llms.txt` index, an `/llms-full.txt` with the main content in one file, and a clean Markdown version of every public page. Build them from the same data and templates the HTML uses, so they never drift from the site.

Site: not given. Find the production URL in the project config, or ask me.

## Ground rules

1. Read before you write. Explore the parts of the codebase this job touches and tell me what is already in place.
2. Plan, then stop. Give me a numbered plan: what you will change, in which files, and why. Wait for my approval before editing anything.
3. Fix things at their source (the shared layout, the config, the generator), not with one-off patches page by page.
4. Keep the project's conventions: its framework APIs, file layout, naming and code style. Add no new dependency without asking.
5. Where something is already done well, say so and leave it alone.

## You will not

- Edit `.env` files, secrets or API keys, or print their values.
- Deploy, push, merge or publish anything.
- Delete data, or run migrations or scripts against a production database.
- Add tracking, third-party scripts or links to any site I did not ask for.
- Change copy, design or behaviour that this job does not need.
- Expose private, signed-in or noindex pages in any of these files.
- Change how the HTML pages look.

## Your stack

- Framework: Work out the framework, router and build output from the project files (package.json, lockfiles, framework config) before planning, and say which you found.
- Host: Work out where the site is hosted from the config files (vercel.json, netlify.toml, wrangler config, Dockerfile, fly.toml) and say which you found.

## Steps

### 1. List the public pages

List every public, indexable page type and where its content comes from (CMS, Markdown files, database, hard-coded components). Exclude anything private, signed-in or `noindex`. The sitemap, if there is one, is usually the right source of truth: reuse it rather than writing a second list.

### 2. Markdown versions of pages

- Serve a Markdown version of each public page at the same URL plus `.md` (the homepage at `/index.md`). Where the framework allows, also answer `Accept: text/markdown` on the normal URL; a request with no Accept header or `*/*` must still get HTML.
- Content pages built from data (posts, docs, product pages) should render Markdown from that data. Hand-written pages can be converted from the page's own main content. Leave out navigation, footers, cookie banners and scripts.
- Each Markdown page starts with `# Title`, a one-line summary as a `>` quote, and the canonical HTML URL. Links are absolute and point at the HTML pages.
- Headers: `Content-Type: text/markdown; charset=utf-8`, `X-Robots-Tag: noindex, follow` (the HTML page stays the one search engines index), and `Link: <html-url>; rel="canonical"`.
- A page with no Markdown version returns a 404 in Markdown, not an HTML page.

### 3. /llms.txt

Follow the format from llmstxt.org:

- `# Site name`, then a `>` quote with one factual sentence on what the site or product is.
- A short paragraph of facts an assistant should know (who it is for, pricing model, what makes it different), with no marketing claims it cannot check.
- `## Sections` of links: `- [Page name](https://site/page.md): what the page holds`. Point at the `.md` versions. Put core pages first and an `## Optional` section last for detail that can be skipped.
- Generate it from the same page list as the sitemap so new pages appear by themselves.

### 4. /llms-full.txt

Join the Markdown of the pages listed in llms.txt into one file, each preceded by a `<!-- source: URL -->` line, most important pages first. Keep it to a size an assistant can read (aim under about 500,000 characters; leave out the least important pages first and say so at the end). Cache it, since it is expensive to build.

### 5. Advertise them

Add `<link rel="alternate" type="text/markdown" href="...md">` to each page's head, and make sure robots.txt does not block `/llms.txt`, `/llms-full.txt` or the `.md` URLs.

## Done means

- [ ] `/llms.txt` follows the llmstxt.org format and links the Markdown versions.
- [ ] `/llms-full.txt` holds the main pages' Markdown with a source line before each.
- [ ] Every public page has a `.md` version with a title, summary and canonical URL, and private pages have none.
- [ ] Markdown responses send `text/markdown`, `noindex, follow` and a canonical Link header.
- [ ] Browsers and normal crawlers still get exactly the same HTML as before.

## Report back

When you finish, give me:

- what you changed, file by file;
- anything you skipped, and why;
- anything I must do by hand (DNS records, dashboards, accounts, deploys);
- how to check each change on the live site.