AI companies back a shared robots.txt training opt-out
Google, Microsoft, OpenAI and Anthropic support a common website signal intended to simplify how publishers reject AI training use.
What happened
Major AI companies are backing a shared policy token for robots.txt that would let websites express AI-training preferences once instead of maintaining separate instructions for each crawler. Google said it plans to interpret the proposed token as a disallow for Google-Extended when no more specific Google-Extended rule is present.
The proposal comes from work in the IETF’s AIPREF process. Google, Microsoft, OpenAI and Anthropic have publicly indicated support, while Meta has also been reported as engaging with the effort. The signal is designed to cover participating providers rather than create another company-specific crawler rule.
Why it matters
The move addresses a practical problem for publishers: today’s opt-out controls differ by provider and may distinguish training, search grounding, and product access. A shared token could lower the technical burden for smaller websites and make publisher preferences easier for crawlers to interpret consistently.
The announcement is not a licensing settlement and does not force every AI company or crawler to comply. It also leaves important questions open, including how providers will document enforcement, how conflicts between broad and provider-specific rules will be resolved, and whether opting out of training will affect search or answer-generation products separately.
For the AI industry, the significance is institutional rather than immediate. If the signal becomes widely implemented, it could become basic web infrastructure for negotiating access to public content. If adoption remains voluntary and uneven, publishers may still need a patchwork of vendor-specific controls.
Uncle Cat take
The useful deliverable is the shared token, but its value depends on crawler compliance records that have not yet been promised.