Key takeaways
- Google, Microsoft, Anthropic and OpenAI said on October 30 they will support a shared robots.txt signal to opt out of AI training, per SiliconANGLE.
- Microsoft's version uses the user agent __usage_ai-training and takes effect February 1, 2027.
- The standard is an IETF Internet-Draft, not final, and Meta has not committed.
Google, Microsoft, Anthropic and OpenAI said on October 30 that they will support a new standard that lets website owners opt out of AI training with a single entry in robots.txt. The AI crawler opt-out standard is described in an Internet-Draft the companies submitted to the IETF, the body that maintains the Robots Exclusion Protocol, according to SiliconANGLE. It is still a draft, so no crawler honors it yet.
Today a publisher who wants out of AI training has to look up each company's crawler name and block them one by one. A shared signal would replace that list with one rule.
What the standard does
SiliconANGLE reports that one robots.txt entry would block every scraper that collects data to train generative AI models. Models that perform only non-generative tasks would not be affected. Meta has said it likes the idea but has not committed to using it, according to the same report.
Microsoft published the most detail. In a post on the Bing blog, it shows the training signal as a robots.txt user agent called __usage_ai-training. Setting Disallow: / under it opts the whole site out. A page can also carry ai-training:disallow in a meta tag or an X-Robots-Tag header, and the page setting overrides robots.txt. The default is Allow.
The shared signal is the IETF draft draft-kelley-aipref-training-preference-signal. Microsoft says the working group's path to a final standard is still ongoing.
Microsoft adds a separate "grounding" setting
Grounding means an AI answer pulls from a page at the moment a user asks, rather than learning from it in training. Microsoft is treating that as its own preference, set per page only. It has three values: Allow, DisallowExtended and Disallow. DisallowExtended keeps a page available to Microsoft's own products such as Copilot and AI Overview but excludes third-party grounding services. Microsoft warns that blocking grounding will reduce a page's visibility in AI-generated answers.
When it starts
SiliconANGLE says Microsoft and OpenAI plan to add support in the coming months and Google sometime next year. Microsoft gives its own dates:
| Date | What happens |
|---|---|
| December 15, 2026 | Temporary grounding preferences open in Bing Webmaster Tools |
| February 1, 2027 | New preferences take effect as Microsoft crawls |
| March 1, 2027 | Deadline to publish page-level preferences |
| September 30, 2027 | Temporary preferences expire |
What to watch
A robots.txt rule only works if crawlers obey it, and the signal controls use of content, not access. Microsoft notes that Allow does not override a robots.txt rule that blocks Bingbot. The test is whether the draft becomes a final standard and whether Meta and smaller AI companies adopt it. We could not read OpenAI's own announcement in this run, so the OpenAI timing here rests on SiliconANGLE's report.