Opting Out of AI Training in One Line: The Shared robots.txt Rule Backed by OpenAI, Google, Microsoft and Anthropic
Until now, if you didn't want your content used to train AI models, you had to block each company's bot one by one in robots.txt: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, Applebot-Extended for Apple and so on. Keeping that list current as new bots appeared was up to you.
On 30 October 2026, OpenAI, Google, Microsoft and Anthropic said they will support a single shared rule instead. It has been submitted as a draft to the Internet Engineering Task Force (IETF), which maintains robots.txt.
How to write the rule
Add these two lines to the robots.txt file at the root of your site:
- User-agent: __usage_ai-training
- Disallow: /
To keep part of the site open to training (a press kit or blog, say), add an allow line:
- User-agent: __usage_ai-training
- Disallow: /
- Allow: /blog/
Who applies it, and when?
- Anthropic (Claude): already honours it.
- Microsoft: reported to start on 1 February 2027.
- Google: reported to start in early 2027.
- OpenAI (ChatGPT): "in the coming months", no firm date.
- Meta: supports the idea but hasn't committed to implementing it.
Until those dates arrive, keep your existing rules: if you block GPTBot, ClaudeBot or Google-Extended, leave those lines in place. OpenAI also says the new signal won't replace its current controls.
The key point: it doesn't turn off search
The rule only covers data collection for training. When ChatGPT, Claude and Perplexity answer a question, they read source pages with separate search bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot). Those bots are not affected. For a business that wants to stay out of model training but still be cited when customers ask ChatGPT, this rule is exactly what it needs.
Blocking search bots instead removes your site from AI answers altogether. When we scanned Turkey's 100 biggest websites, five news sites were also blocking search bots, so their content can't be cited in ChatGPT or Perplexity answers.
Limits to keep in mind
- It states a preference, not a technical barrier: compliance with robots.txt is voluntary.
- It isn't a formal standard yet; it is a draft at the IETF.
- It doesn't reach back to data already collected; it only applies going forward.
- It only covers generative AI training; search indexing and real-time retrieval follow separate rules.
Which choice is right for you?
- Local business, clinic, salon or hotel: staying open to training probably helps; the better AI knows you, the more accurately it recommends you.
- News site, publisher or original content creator: closing training while keeping search open is a sensible middle ground.
- Either way: don't block the search bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot).
See which bots your robots.txt allows, and whether it uses the new shared rule, with our free site scan. Sources: SiliconANGLE, Engadget.