Most advice on how to block AI crawlers with robots.txt is a list of names to paste in. The operators' own documentation shows why that is not enough. One company can run an agent that gathers training data, another that builds a search index for its assistant, and a third that fetches a page because a user asked. Block all three and you opt out of training, but you also drop out of the answers that might cite and link to you.
This guide reads the crawler pages of OpenAI, Anthropic, Google, Apple and Common Crawl, RFC 9309 and the IETF's AI preferences drafts, all checked on 4 November 2026. It is a reading of published documents, not legal advice.
Three jobs, three kinds of token
Read side by side, the vendor pages describe three purposes, each with its own token: training (GPTBot, ClaudeBot), search inside an AI product (OAI-SearchBot, Claude-SearchBot) and user-triggered fetches (ChatGPT-User, Claude-User).[1][2] OpenAI states the rule that matters most: "Each setting is independent of the others", so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot.[1] Anthropic lists a separate consequence for disabling each of its three bots.[2]
Two tokens are not crawlers at all. Google says Google-Extended "doesn't have a separate HTTP request user agent string" and that its token "is used in a control capacity."[3] Apple says "Applebot-Extended does not crawl webpages" and is "only used to determine how to use the data crawled by the Applebot user agent."[6] Our reading is that disallowing these two changes how collected data may be used, not which requests reach your server.
The documented tokens at a glance
The table records what each operator's page said on 4 November 2026.
| Token | Operator | Documented purpose | Effect of disallowing, as documented |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's generative AI foundation models | Indicates the content "should not be used in training generative AI foundation models"[1] |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | The site "will not be shown in ChatGPT search answers", though it can still appear as navigational links[1] |
| ChatGPT-User | OpenAI | Visits pages for user actions in ChatGPT and Custom GPTs; not automatic crawling | "robots.txt rules may not apply" because a user initiated the action[1] |
| ClaudeBot | Anthropic | Collects web content that could contribute to model training | Signals that the site's "future materials" should be excluded from training datasets[2] |
| Claude-SearchBot | Anthropic | Analyses content to improve search results for users | Stops indexing for search, which "may reduce your site's visibility and accuracy in user search results"[2] |
| Claude-User | Anthropic | Retrieves pages when a user asks Claude a question | Stops retrieval for user queries, which "may reduce your site's visibility for user-directed web search"[2] |
| Googlebot | Crawls for Google Search and other Google products | Affects Google Search "including Discover and all Google Search features"[3] | |
| Google-Extended | Control token for training future Gemini models and for grounding in Gemini Apps and Vertex AI | Does not affect inclusion in Google Search and is not a ranking signal[3] | |
| Applebot | Apple | Search in Spotlight, Siri and Safari; its data may also help train Apple foundation models | Allowing it is what lets content appear in Apple's search results[6] |
| Applebot-Extended | Apple | Control token for training Apple's foundation models | Opts out of that training; such pages "can still be included in search results"[6] |
| CCBot | Common Crawl | Crawls for Common Crawl's open repository of web crawl data | Prevents Common Crawl from crawling the site[7] |
An example robots.txt that keeps search and opts out of training
This file uses only tokens from the table. It allows every crawler by default, then gives each training token its own group.
# robots.txt for https://www.example.com
# Tokens checked against each operator's documentation on 2026-11-04.
# Default for every crawler not named below: crawl everything.
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# Googlebot and Applebot fall through to this group.
User-agent: *
Allow: /
# OpenAI: GPTBot crawls content that may be used to train
# OpenAI's generative AI foundation models.
User-agent: GPTBot
Disallow: /
# Anthropic: ClaudeBot collects content that could contribute to
# model training. A block signals that future material should be
# excluded from training datasets.
User-agent: ClaudeBot
Disallow: /
# Google: Google-Extended is a control token, not a crawler. It
# covers Gemini training and grounding in Gemini Apps and Vertex AI.
# Google says it does not affect Google Search inclusion or ranking.
User-agent: Google-Extended
Disallow: /
# Apple: Applebot-Extended does not crawl. It governs whether data
# crawled by Applebot is used to train Apple's foundation models.
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlCCBot is left out on purpose: Common Crawl describes an open repository "universally accessible and analyzable by anyone" and states no training purpose of its own.[7] Our reading is that blocking it is a decision about a public archive; its rule is User-agent: CCBot with Disallow: /.
How crawlers read the file
- A named group replaces the default. Crawlers use "case-insensitive matching to find the group that matches the product token" and fall back to
*only "If no matching group exists."[8] Repeat any shared rule, such asDisallow: /admin/, in every named group. - One file per host. The file lives at "scheme:[//authority]/robots.txt", and Anthropic asks owners to add rules "for every subdomain that you wish to opt out from."[8][2]
- Status codes change the meaning. On a 4xx response a crawler "MAY access any resources on the server"; on a 5xx it "MUST assume complete disallow."[8]
- Changes take time. Crawlers "SHOULD NOT use the cached version for more than 24 hours", and OpenAI says search results take about 24 hours to adjust.[8][1]
- Apple borrows Googlebot rules. "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions."[6]
The trade-offs, in our reading
- Blocking search and fetch agents costs visibility. OpenAI and Anthropic both say so.[1][2] If citations and referral clicks from assistants matter, keep those agents allowed and decide on training separately.
- Google bundles training and grounding. One token covers both.[3] Google's blog adds that "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."[5] Our reading is that a Google-Extended block is likely to cost Gemini Apps visibility too; the pages read here describe no token that separates the two.
- AI Overviews and AI Mode sit under Search. Google says "robots.txt directives for Googlebot is the control" for AI features in Search, with
nosnippet,data-nosnippet,max-snippetandnoindexto limit what is shown.[4] Our reading: no robots.txt token keeps a page in Google Search but out of those features. - Apple splits training from answers. Applebot-Extended handles training; for "broad world knowledge answers" in Siri and Search, Apple points to the
nosnippetmeta tag.[6]
What RFC 9309 says robots.txt is
RFC 9309 (September 2022) is an IETF Proposed Standard. It defines rules that "crawlers are requested to honor" and says plainly: "These rules are not a form of access authorization." The protocol "is not a substitute for valid content security measures", listing paths "exposes them publicly", and HTTP authentication is the suggested alternative. It also rests on self-identification: "Crawlers set their own name, which is called a product token, to find relevant groups."[8] Every rule above works only if the crawler announces the matching name and chooses to obey.
What robots.txt cannot do
Reach content already collected. Anthropic frames a ClaudeBot block around "future materials".[2] The IETF attachment draft says "updates to 'robots.txt' do not retroactively apply to resources."[11] None of the crawler pages read here describes removing data already gathered; ask each vendor.
Stop crawlers that lie about who they are. Google warns that "The HTTP user agent string can be spoofed", and Common Crawl is "aware of crawlers falsely identifying themselves as CCBot."[3][7]
Bind every user-triggered fetch. For ChatGPT-User, OpenAI says "robots.txt rules may not apply"; Anthropic says Claude-User lets owners control access through user-initiated requests.[1][2] The two vendors differ.
Enforce anything. The aipref vocabulary draft does not "ensure that preferences are followed", and a recipient "has a choice whether to follow those preferences."[10] The working group's charter puts "Technical enforcement of preferences" out of scope.[9]
EU policy adds some weight. In the copyright chapter of the General-Purpose AI Code of Practice, signatories that crawl for text and data mining and model training commit "to employ web-crawlers that read and follow instructions expressed in accordance with the Robot Exclusion Protocol (robots.txt)" as specified in RFC 9309.[13] The Commission calls the code "a voluntary tool" and lists Anthropic, Google and OpenAI among its signatories.[14] For the wider rules, see what general-purpose AI model providers must do under the EU AI Act.
Alternatives, and their limits
- Verify, then block at the server. OpenAI publishes gptbot.json, searchbot.json and chatgpt-user.json; Anthropic bots.json; Google common-crawlers.json plus a reverse DNS pattern; Apple an Applebot IP list; Common Crawl ccbot.json.[1][2][3][6][7]
- Keep robots.txt reachable when you block. Anthropic warns that IP blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."[2] Our reading: keep /robots.txt reachable when blocking at the edge, because under RFC 9309 a 4xx answer tells a crawler there are no rules.[8]
- Authenticate private content, as the RFC recommends.[8]
- Use page-level controls for search-side AI features: Google's snippet controls, Apple's
nosnippet.[4][6] - Contracts and law. The EU code links robots.txt to rights reservations under Article 4(3) of Directive (EU) 2019/790 and says it "does not affect the right of rightsholders to expressly reserve the use of works" in "any appropriate manner".[13] Our reading is that terms of use may form part of a reservation, but whether they are effective is a legal question this guide does not answer.
The IETF aipref work: a shared signal, not yet a standard
The IETF's AI Preferences working group is chartered to "standardize building blocks that allow for the expression of preferences about how content is collected and processed" for AI, including attachment through robots.txt and HTTP header fields. Its milestones targeted August 2026 for sending both core documents to the IESG.[9]
| Document | What it proposes | Datatracker status, 4 November 2026 |
|---|---|---|
| draft-ietf-aipref-vocab-08 | Usage categories: AI Training (train-ai), AI Use (ai-use), Search (search) | Active Internet-Draft, WG Document, intended Proposed Standard, IESG state "I-D Exists"[10] |
| draft-ietf-aipref-attach-05 | A Content-Usage rule for robots.txt and a Content-Usage HTTP header field | Active Internet-Draft, WG Document, intended Proposed Standard, IESG state "I-D Exists"[11] |
| draft-kelley-aipref-training-preference-signal-00 | A cross-vendor __usage_ai-training policy token written as an ordinary robots.txt group | Active individual Internet-Draft, no stream[12] |
Neither working-group draft is an RFC, and the vocabulary draft says its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part." It defines AI Training as "Using an asset to modify the learned parameters of a generative AI model."[10] The attachment draft separates acquisition (Allow and Disallow) from usage, so a file could allow crawling and still state Content-Usage: train-ai=n.[11]
The individual draft, whose author is listed with an Anthropic affiliation, adds no directive, only a group named by a policy token. The * group "MUST NOT be interpreted as matching the policy token", and the token "is not a directive for crawling behavior."[12]
# Proposed in an individual Internet-Draft (October 2026), not a standard.
User-agent: __usage_ai-training
Disallow: /Google's crawler page says it "intends to support" this signal in early 2027.[3] Its blog says that when a site uses the token to opt out "and Google-Extended is not explicitly specified", Google plans to treat it as a Google-Extended disallow, and that "Existing company-specific controls will continue to function alongside this new joint signal."[5] Parsers that do not implement the draft are meant to ignore the group.[12] Our reading: the text may still change, so keep the vendor groups until support is confirmed.
A checklist
- Decide per purpose: training, AI search, user-triggered fetches.
- Name only tokens the operator documents, and re-read its page before each change.
- Serve one robots.txt per host, as UTF-8
text/plain, and repeat shared rules in every named group.[8] - Check log entries against published IP lists before trusting a user agent.
- Put anything private behind authentication, and revisit in 2027 when the shared token may be supported.
For the reverse question, what vendors keep from the prompts you send, see our AI vendor data retention checklist. For why retrieval-time access shapes AI answers, see how RAG retrieval works. More in policy and developer tools.
