Should I block AI crawlers?

It is a business decision. Blocking training bots keeps your text out of future models; blocking search-style bots can remove you from AI answers that send visitors. The page shows both groups separately so you can choose.

AI crawler policy checker

Which AI bots your robots.txt allows, and whether you publish llms.txt.

What this check does

Reads robots.txt and reports the policy for two dozen known AI user agents (training, search and user-initiated fetchers from OpenAI, Anthropic, Google, Apple, Common Crawl, Perplexity, ByteDance, Meta and others), then checks for llms.txt and llms-full.txt and whether they follow the convention.

Per-agent policy: allowed, partial or blocked, and which group decided it
Training versus search versus user-initiated agents
llms.txt presence, title, summary, sections and links
Sitemap declaration

Questions

Should I block AI crawlers?: It is a business decision. Blocking training bots keeps your text out of future models; blocking search-style bots can remove you from AI answers that send visitors. The page shows both groups separately so you can choose.