Training, search and user-triggered crawlers
AI companies run different bots for different jobs. Training crawlers such as GPTBot and ClaudeBot collect pages that may be used to train models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so an assistant can cite and link them. User-triggered agents such as ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks about it.
Because each has its own robots.txt name, you can block training and still allow search. Google-Extended and Applebot-Extended are names only, not crawlers: they control whether pages that Google's and Apple's own crawlers fetch may be used for AI.
How the check reads robots.txt
It follows the robots.txt standard (RFC 9309). A crawler obeys the group of rules that names it; if no group does, it obeys the * group. Within that group, the longest matching Allow or Disallow rule wins, and Allow wins a tie.
If robots.txt returns a 4xx error such as 404, crawlers may fetch everything. If it returns a server error or doesn't answer, crawlers treat the whole site as blocked. The check covers your home page; rules for individual folders can differ.
What llms.txt adds
llms.txt is a Markdown file at /llms.txt that summarises your site for AI assistants, starting with a # title line. It is a proposed convention, not a standard that search engines have committed to, and it doesn't control crawling.
The checker confirms the file exists, is served as text rather than an HTML page and starts with a # title.