robots.txt for AI crawlers
Which AI user agents exist, what each one is for, and why blocking the wrong one removes you from the answers instead of protecting you.
Most robots.txt advice about AI is written for publishers, whose content is the product and who lose when it is summarised. A developer tool is in the opposite position: it wants to be read, quoted and installed.
Getting this file wrong in that direction is expensive and easy.
The distinction that matters
A training crawler collects pages that may be used to train future models. A retrieval crawler fetches a page because somebody asked a question right now. Blocking a retrieval crawler removes you from answers this week. Blocking a training crawler changes what models know about you in years to come.
Almost everybody conflates the two, blocks GPTBot, and believes they have opted out of AI. They have opted out of one thing and left the other running.
The user agents, and what each one is
| User agent | Operator | What it does |
|---|---|---|
GPTBot | OpenAI | Training crawler |
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results |
ChatGPT-User | OpenAI | Fetches a page because a user's question needs it now |
ClaudeBot | Anthropic | Training crawler |
Claude-User | Anthropic | Fetches a page for a user's request now |
Claude-SearchBot | Anthropic | Indexes pages for search results |
PerplexityBot | Perplexity | Indexes pages for answers |
Perplexity-User | Perplexity | Fetches a page a user's question needs |
Google-Extended | Controls use in Gemini and grounding, not classic Search indexing | |
GoogleOther | Research and product crawls | |
Applebot-Extended | Apple | Controls training use; Applebot itself serves Siri and Spotlight |
Bytespider | ByteDance | Training crawler |
CCBot | Common Crawl | Open crawl that many datasets derive from |
Amazonbot, meta-externalagent, cohere-ai, Diffbot, YouBot, Mistralai-User, DuckAssistBot | various | A mix of training and retrieval |
Two traps in that table.
Google-Extended is not a search crawler. Blocking it does not remove you from Google Search. It removes you from Gemini's grounding. People block it thinking they are protecting search rankings and instead remove themselves from an answer surface.
Applebot and Applebot-Extended are different. The first serves Siri and Spotlight. The second controls training use. Blocking the wrong one has the wrong effect.
What a developer tool should publish
Allow everything, and protect the specific things that need protecting.
# Everything is welcome to read the site.
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlThat is the whole file for most developer tools. The long version, naming each agent explicitly, has one real advantage: it survives a future edit that adds a blanket Disallow for a different reason. If you name the agents, a later change is less likely to remove them by accident.
What to actually protect
robots.txt is a request, not an access control. It is the wrong tool for anything that matters.
| Concern | Wrong tool | Right tool |
|---|---|---|
| Private data | robots.txt | Authentication |
| Expensive endpoints | robots.txt | Rate limiting, a paid tier |
| Bulk scraping of a data set | robots.txt | Serve it behind a function with a token |
| Draft pages | robots.txt | Do not publish them |
Listing a sensitive path in robots.txt publishes the path. It is a public file.
The one case for blocking
If your content is the product, blocking training crawlers is a real business decision. A publisher, a paid research site, a documentation product sold by subscription.
Even then, keep the retrieval crawlers allowed unless you have decided you do not want to appear in answers at all. Those two choices are separable and most people make them by accident together.
What this does not solve
Allowing the crawlers gets you read. It does not get you chosen.
In our measurements, on decision tasks Claude Code ran a web search in 1.6% of runs and Codex in 53%. So for a large share of the sessions where a coding agent picks a product, no crawler was involved at all and the decision came from the repository and from what the model already knew.
robots.txt is a five-minute check that removes an own goal. It is not a strategy. See agent discoverability for what is.
The check
curl -s https://example.com/robots.txtRead it and ask three questions. Is anything I want in AI answers disallowed? Is Google-Extended blocked by somebody who thought it was a search crawler? Does the file list a path I would rather nobody knew about?
Common questions
Which AI crawlers should I allow in robots.txt?
If you want to appear in AI answers, allow the retrieval and search agents: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, Google-Extended and Applebot-Extended. Training crawlers such as GPTBot, ClaudeBot, CCBot and Bytespider are a separate decision.
What is the difference between a training crawler and a retrieval crawler?
A training crawler collects pages that may be used to train future models. A retrieval crawler fetches a page because a user asked a question right now. Blocking the second removes you from answers immediately; blocking the first affects what models learn over years.
Does blocking GPTBot stop ChatGPT from mentioning me?
Not directly. GPTBot is the training crawler. The agents that fetch pages to answer a live question are OAI-SearchBot and ChatGPT-User, and blocking those is what removes you from answers.
Should a developer tool block AI crawlers?
Almost never. A developer tool wants to be installed by agents, and being absent from what they can read is the opposite of that. The case for blocking applies to sites whose content is the product, such as publishers.
Where this comes from
Armature ran 5,292 judged sessions with Claude Code, Codex and Cursor inside 51 realistic codebases, and published every run. The numbers on this page come from that work.