# robots.txt for AI crawlers > Which AI user agents exist, what each one is for, and why blocking the wrong one removes you from the answers instead of protecting you. Source: https://armature.tech/library/robots-txt-for-ai-crawlers Published: 2026-09-03 Publisher: Armature, Inc. (https://armature.tech) --- Most `robots.txt` advice about AI is written for publishers, whose content is the product and who lose when it is summarised. A developer tool is in the opposite position: it wants to be read, quoted and installed. Getting this file wrong in that direction is expensive and easy. ## The distinction that matters > A **training crawler** collects pages that may be used to train future models. A **retrieval crawler** fetches a page because somebody asked a question right now. Blocking a retrieval crawler removes you from answers this week. Blocking a training crawler changes what models know about you in years to come. Almost everybody conflates the two, blocks GPTBot, and believes they have opted out of AI. They have opted out of one thing and left the other running. ## The user agents, and what each one is | User agent | Operator | What it does | | --- | --- | --- | | `GPTBot` | OpenAI | Training crawler | | `OAI-SearchBot` | OpenAI | Indexes pages for ChatGPT search results | | `ChatGPT-User` | OpenAI | Fetches a page because a user's question needs it now | | `ClaudeBot` | Anthropic | Training crawler | | `Claude-User` | Anthropic | Fetches a page for a user's request now | | `Claude-SearchBot` | Anthropic | Indexes pages for search results | | `PerplexityBot` | Perplexity | Indexes pages for answers | | `Perplexity-User` | Perplexity | Fetches a page a user's question needs | | `Google-Extended` | Google | Controls use in Gemini and grounding, not classic Search indexing | | `GoogleOther` | Google | Research and product crawls | | `Applebot-Extended` | Apple | Controls training use; `Applebot` itself serves Siri and Spotlight | | `Bytespider` | ByteDance | Training crawler | | `CCBot` | Common Crawl | Open crawl that many datasets derive from | | `Amazonbot`, `meta-externalagent`, `cohere-ai`, `Diffbot`, `YouBot`, `Mistralai-User`, `DuckAssistBot` | various | A mix of training and retrieval | Two traps in that table. **`Google-Extended` is not a search crawler.** Blocking it does not remove you from Google Search. It removes you from Gemini's grounding. People block it thinking they are protecting search rankings and instead remove themselves from an answer surface. **`Applebot` and `Applebot-Extended` are different.** The first serves Siri and Spotlight. The second controls training use. Blocking the wrong one has the wrong effect. ## What a developer tool should publish Allow everything, and protect the specific things that need protecting. ``` # Everything is welcome to read the site. User-agent: * Allow: / Disallow: /api/ Sitemap: https://example.com/sitemap.xml ``` That is the whole file for most developer tools. The long version, naming each agent explicitly, has one real advantage: it survives a future edit that adds a blanket `Disallow` for a different reason. If you name the agents, a later change is less likely to remove them by accident. ## What to actually protect `robots.txt` is a request, not an access control. It is the wrong tool for anything that matters. | Concern | Wrong tool | Right tool | | --- | --- | --- | | Private data | `robots.txt` | Authentication | | Expensive endpoints | `robots.txt` | Rate limiting, a paid tier | | Bulk scraping of a data set | `robots.txt` | Serve it behind a function with a token | | Draft pages | `robots.txt` | Do not publish them | Listing a sensitive path in `robots.txt` publishes the path. It is a public file. ## The one case for blocking If your content is the product, blocking training crawlers is a real business decision. A publisher, a paid research site, a documentation product sold by subscription. Even then, keep the retrieval crawlers allowed unless you have decided you do not want to appear in answers at all. Those two choices are separable and most people make them by accident together. ## What this does not solve Allowing the crawlers gets you read. It does not get you chosen. In our measurements, on decision tasks Claude Code ran a web search in 1.6% of runs and Codex in 53%. So for a large share of the sessions where a coding agent picks a product, no crawler was involved at all and the decision came from the repository and from what the model already knew. `robots.txt` is a five-minute check that removes an own goal. It is not a strategy. See [agent discoverability](/library/agent-discoverability) for what is. ## The check ```bash curl -s https://example.com/robots.txt ``` Read it and ask three questions. Is anything I want in AI answers disallowed? Is `Google-Extended` blocked by somebody who thought it was a search crawler? Does the file list a path I would rather nobody knew about? ## Common questions ### Which AI crawlers should I allow in robots.txt? If you want to appear in AI answers, allow the retrieval and search agents: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, Google-Extended and Applebot-Extended. Training crawlers such as GPTBot, ClaudeBot, CCBot and Bytespider are a separate decision. ### What is the difference between a training crawler and a retrieval crawler? A training crawler collects pages that may be used to train future models. A retrieval crawler fetches a page because a user asked a question right now. Blocking the second removes you from answers immediately; blocking the first affects what models learn over years. ### Does blocking GPTBot stop ChatGPT from mentioning me? Not directly. GPTBot is the training crawler. The agents that fetch pages to answer a live question are OAI-SearchBot and ChatGPT-User, and blocking those is what removes you from answers. ### Should a developer tool block AI crawlers? Almost never. A developer tool wants to be installed by agents, and being absent from what they can read is the opposite of that. The case for blocking applies to sites whose content is the product, such as publishers. ## Read next - [llms.txt: what it is, what it does, and what to put in it](https://armature.tech/library/llms-txt-guide) (Markdown: https://armature.tech/library/llms-txt-guide.md) - [Audit your documentation for coding agents](https://armature.tech/library/audit-your-docs-for-coding-agents) (Markdown: https://armature.tech/library/audit-your-docs-for-coding-agents.md) - [Documentation for coding agents](https://armature.tech/library/documentation-for-coding-agents) (Markdown: https://armature.tech/library/documentation-for-coding-agents.md) - [Generative engine optimization (GEO)](https://armature.tech/library/generative-engine-optimization) (Markdown: https://armature.tech/library/generative-engine-optimization.md) --- Armature helps software products get discovered and used by coding agents. Service: https://armature.tech/discoverability · Results: https://armature.tech/leaderboards/sectors · Contact: contact@armature.tech