AI crawlers, robots.txt and llms.txt

Three different kinds of automated visitor now carry "AI" labels, and they deserve different decisions.

Three kinds of bot

• Training crawlers collect pages to build future models: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), Applebot-Extended, CCBot (Common Crawl, whose archive feeds many models), Bytespider, meta-externalagent. Blocking these keeps your text out of models trained after the block; it does not remove anything already collected.
• Search and answer crawlers index pages so an AI assistant can cite and link to them: OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking these removes you from AI answers that would otherwise send visitors.
• User-initiated fetchers load a page because a person asked the assistant about it: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User. These behave like a browser acting for one user and generally fetch one page at a time.

Controlling them

All of the reputable ones honour robots.txt. A group per agent is the clearest form:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /

A rule for User-agent: * applies to any agent that has no group of its own, which is why a blanket Disallow: /account/ shows every AI agent as "partial" on the policy table: they are allowed in, just not everywhere. Agents that ignore robots.txt need firewall or CDN rules instead; the checker cannot see those.

llms.txt

/llms.txt is a short Markdown file that tells a language model what a site is and where its useful pages are: an H1 with the site name, a one-line blockquote summary, then sections of links with a sentence each. /llms-full.txt optionally carries the full text of the key pages. Neither is a standard with enforcement; they are a courtesy that some assistants read when a user asks about your site. The checker confirms the file is served as text, not an HTML error page, and that it has the expected shape.

Deciding

A business that sells through search traffic usually wants search-style agents allowed and may or may not mind training. A publisher with licensing concerns usually blocks training agents. A site with an llms.txt is saying "here is how to summarise me correctly". The policy checker shows each group separately so the decision can be made deliberately rather than by default.