🔍 Read the full analysis: AI Training Opt-Outs Get A Shared Approach From Four Major Companies on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared robots.txt signal asking that a website’s content not be used to train generative AI models. The signal is voluntary, does not cover AI answers based on live web content, and will roll out on different schedules; Meta expressed support but made no commitment to implement it.
OpenAI, Google, Microsoft and Anthropic said on October 30 they will support a shared robots.txt signal that asks companies not to use a website’s content to train generative AI models. The move could let publishers state that preference once instead of maintaining separate training rules for each company, but the signal is voluntary and the firms have set different implementation timelines.
The proposed rule uses the user-agent name __usage_ai-training followed by Disallow: /. It is set out in an Internet-Draft titled “Machine-Readable Training Preference Signal,” published by Kevin Kelley of Anthropic on October 30. Anthropic says it has already started implementing support, while OpenAI says it aims to do so “in the coming months.”
Microsoft will honor the training preference from February 1, 2027, as it crawls websites. Google says it aims to support the signal in 2027; its crawler documentation specifies “early 2027.” Google says that when a site uses the shared token but has no separate rule for Google-Extended, it plans to treat the shared signal as a disallow for Google-Extended.
Microsoft also announced a separate page-level preference for AI Grounding, covering web content used to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both new preferences take effect February 1, 2027. Its existing NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027. The companies’ existing crawler-specific controls, including GPTBot, Google-Extended and ClaudeBot, continue to apply during the changeover.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Several Company Timelines
A shared signal could reduce the work required for publishers who want to express a training preference to multiple AI providers. Instead of relying only on separate instructions for each crawler, a site could publish one rule for participating companies to recognize. That may make the preference easier to maintain, particularly for organizations managing many websites.
But the change does not create a technical barrier that prevents a crawler from accessing a site. It is a request that participating companies say they will honor, not an enforcement mechanism. It also does not establish compensation for publishers or promise that opting out will affect how often a site appears in search results or AI products. Microsoft stated that neither of its preferences guarantees inclusion, traffic or payment.
The distinction between training and AI-generated answers also matters. A publisher may want to prevent its material from being used to train a model while still allowing it to inform an answer based on current web content, or may want to restrict both. Microsoft’s separate grounding setting makes that distinction explicit for its services; the shared training token alone does not cover live-content answers.
Products Worth Considering
Levoit Vital 100S-P Air Purifier with Washable Pre-Filter
Therabody Theragun Elite 5th Gen Deep Tissue Massage Gun
Withings BPM Connect Wi-Fi and Bluetooth Blood Pressure Monitor
As an affiliate, we earn on qualifying purchases.
A Draft, Not an Internet Standard
The proposal is connected to the Internet Engineering Task Force’s AI Preferences working group, which discusses technical approaches for expressing preferences about web content. The document is an individual Internet-Draft, not an adopted IETF standard. At a working-group meeting on October 21, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” The draft therefore records a proposal and does not, by itself, bind companies or establish a formal internet-wide rule.
The companies’ statements differ in specificity. Anthropic’s Kelley said the company had started implementation, while adding that it would take time to extend support across its services. OpenAI described its plan as a “voluntary effort.” Microsoft gave a start date, and Google set a 2027 target. Meta spokesperson Andy Stone said the company strongly supports publishers’ ability to express preferences and called a common signal an important step, but did not say Meta would honor it.
“Meta “strongly support[s] publishers’ ability to express preferences””
— Meta spokesperson Andy Stone
Adoption and Enforcement Remain Open
Meta’s implementation plans are unconfirmed, and the source material identifies no public commitment from Apple, Amazon, xAI or Perplexity. It is not clear how many other AI providers will adopt the rule, how consistently participating companies will apply it across their services, or how they will respond if a site’s instructions conflict.
The signal does not block crawling, remove material from models that have already been trained, or prevent use of content for live AI answers. Its status is also unsettled within the IETF process: the draft has no formal standing as an adopted standard. Publishers should not treat the shared line as a guaranteed technical safeguard or assume that every company will recognize it.
Publishers Face a Staged Changeover
Website operators can add the shared training rule if they want participating companies to treat their content as off-limits for model training. They should keep existing company-specific crawler rules in place until each provider’s support is active, since rollout dates differ and the older controls remain relevant during the transition period.
Microsoft users of NoArchive or NoCache should review their settings ahead of March 1, 2027, when Microsoft says those tags will stop counting as training or grounding preferences. Google’s stated target is early 2027, and OpenAI has not given a specific calendar date beyond “the coming months.” The next developments to watch are actual implementation across company services, any further provider commitments, and whether the IETF working group reaches consensus on the draft.
Key Questions
What does the shared AI training opt-out do?
It gives website owners a common robots.txt instruction asking participating companies not to use the site’s content to train generative AI models. It does not technically prevent a crawler from visiting the site.
Which companies have said they will honor it?
OpenAI, Google, Microsoft and Anthropic have announced support, with different timelines. Meta expressed support for the idea but did not commit to implementing it.
Does the rule block content from AI answers?
No. The shared token addresses model training, not AI answers built from live web content. Microsoft announced a separate AI Grounding preference for content used in answers from Bing, Copilot and Microsoft 365.
Is this an official internet standard?
No. The signal is described in an Internet-Draft associated with the IETF’s AI Preferences working group. The draft has not been adopted as a formal IETF standard.
When should publishers update their settings?
They can add the shared training rule now, but should retain company-specific crawler rules until providers implement the new signal. Microsoft says its new preferences begin February 1, 2027, and its NoArchive and NoCache tags stop counting for training or grounding preferences after March 1, 2027.
Source: ThorstenMeyerAI.com
