What belongs in robots.txt
robots.txt tells crawlers which paths they may fetch. Block pages that should never be crawled, like admin panels, account pages and internal search results, and point crawlers to your sitemap.
Do not block CSS or JavaScript files; search engines need them to render the page. And remember that robots.txt is public: it is not a way to hide secret URLs.
AI crawlers
OpenAI uses GPTBot, Anthropic uses ClaudeBot, Perplexity uses PerplexityBot, and Google uses the Google-Extended token for AI training without affecting Search. Allowing them lets your pages be read and cited; blocking them is a business decision.
Questions
Does robots.txt remove pages from Google?
No. It stops crawling. Use a noindex robots tag on the page to keep it out of results.
Does blocking Google-Extended affect my rankings?
No. Google says Google-Extended controls use for its AI models and does not affect Google Search.
Where should I upload the file?
To the root of the host: https://yoursite.com/robots.txt.
How do I check the file works?
Use the robots.txt tester on any page of your site to see what each crawler may fetch.