How robots.txt rules are read
Under the robots.txt standard (RFC 9309), a crawler uses the group that names it, or the * group if none does. Within that group the longest matching rule wins, and Allow wins a tie. A * in a path matches anything and a $ marks the end of the URL.
AI companies use their own crawler names, such as GPTBot for OpenAI and ClaudeBot for Anthropic, so a rule for * does not tell the whole story. Many sites block or allow them on purpose.
Blocking is not the same as removing
robots.txt stops crawling, not indexing. A blocked page can still appear in results if other sites link to it, shown without a description. To keep a page out of search results, allow crawling and add a noindex robots tag instead.
Questions
Where does robots.txt have to be?
At the root of the host, for example https://yoursite.com/robots.txt. Each subdomain needs its own.
Should I block AI crawlers?
It depends on whether you want AI assistants to read and cite your pages. For most startups being cited is useful, so allowing them is common; blocking is a business choice.
What happens if there is no robots.txt?
Crawlers treat everything as allowed. That is fine for most small sites.
Why is my page blocked even though I only disallowed a folder?
A rule like Disallow: /app also matches /apple, because rules match by prefix. End folder rules with a slash, such as /app/.