A sitemap that stays correct, plus IndexNow

Paste into any coding agent: Claude Code, Cursor, Codex, Copilot, Windsurf or Cline.

# A sitemap that stays correct, plus IndexNow

A sitemap tells search engines which pages of the production site exist and when they changed. Hand-written or stale sitemaps list pages that redirect, miss new ones, and stamp every URL with today's date, which teaches crawlers to ignore the dates. Generate it from the same routes and data the site serves, and tell search engines about changes as they happen with IndexNow.

Site: not given. Find the production URL in the project config, or ask me.

## Ground rules

1. Read before you write. Explore the parts of the codebase this job touches and tell me what is already in place.
2. Plan, then stop. Give me a numbered plan: what you will change, in which files, and why. Wait for my approval before editing anything.
3. Fix things at their source (the shared layout, the config, the generator), not with one-off patches page by page.
4. Keep the project's conventions: its framework APIs, file layout, naming and code style. Add no new dependency without asking.
5. Where something is already done well, say so and leave it alone.

## You will not

- Edit `.env` files, secrets or API keys, or print their values.
- Deploy, push, merge or publish anything.
- Delete data, or run migrations or scripts against a production database.
- Add tracking, third-party scripts or links to any site I did not ask for.
- Change copy, design or behaviour that this job does not need.
- Submit anything to Google Search Console or Bing Webmaster Tools on my behalf.

## Your stack

- Framework: Work out the framework, router and build output from the project files (package.json, lockfiles, framework config) before planning, and say which you found.
- Host: Work out where the site is hosted from the config files (vercel.json, netlify.toml, wrangler config, Dockerfile, fly.toml) and say which you found.

## Steps

### 1. Compare the sitemap with reality

Find how the sitemap is built today. List URLs in it that redirect, return 404, are `noindex`, need sign-in, or have a different canonical, and public pages that are missing. Show me the counts.

### 2. Generate it from the source

- Build the sitemap from the route list and the database or content files, with the same rules the pages use to decide whether they are indexable (a list page with too few items that is `noindex` must not be in the sitemap either).
- `lastmod` is the real date the page's content changed (the post's updated date, the newest item on a list page), never the build time. Leave it out rather than invent it.
- Use canonical URLs only. Split into a sitemap index if it passes 50,000 URLs or 50 MB.
- Reference it from robots.txt with a `Sitemap:` line.

### 3. IndexNow

Add IndexNow: generate a key, serve the key file at `/{key}.txt`, and when content is published, updated or deleted, send the changed URLs to `https://api.indexnow.org/indexnow` (in batches, after the change is live, never on every request). Keep the key in configuration, not hard-coded. Bing, Yandex and other engines use it; Google does not, and relies on the sitemap.

### 4. Guard it

Add a test that the sitemap contains no URL that is `noindex`, private or redirected, and that `lastmod` values come from content dates.

## Done means

- [ ] The sitemap is generated from the real routes and data.
- [ ] It holds no redirected, 404, noindex, private or non-canonical URLs.
- [ ] lastmod is the content's real change date, or absent.
- [ ] robots.txt points at the sitemap.
- [ ] IndexNow pings changed URLs after publishing, with the key file served.

## Report back

When you finish, give me:

- what you changed, file by file;
- anything you skipped, and why;
- anything I must do by hand (DNS records, dashboards, accounts, deploys);
- how to check each change on the live site.