Tech Giants Band Together on New Standard for AI Scraping Opt-Outs

This new control enables publishers to apply their AI training preferences across multiple crawlers simultaneously.

OpenAI, Google, Microsoft, and Anthropic have announced their intention to adopt a new standard that allows publishers to opt out of AI scraping, as previously reported by Search Engine Roundtable. This development enables publishers to set their preferences for AI training once, rather than configuring them individually for each AI company.

The proposed shared “policy token” signal was created by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). This signal will reside within a website’s robots.txt file, a mechanism historically used to signal which web crawlers are permitted to scrape a site for search indexing, archiving, or AI training. Currently, websites must signal their AI training preferences separately to each AI company.

As publishers continue to face the influx of AI crawlers scraping their content for model training, this announcement addresses a pressing issue. Various organizations have attempted to combat unauthorized scraping, such as with the Really Simple Licensing (RSL) standard, which charges crawlers, and Cloudflare’s default blocking of AI crawlers.

OpenAI, Google, Microsoft, and Anthropic have stated that their AI crawlers will be restricted from accessing a site if they encounter this shared token. Microsoft is taking a more granular approach, offering publishers the ability to opt out of AI training or AI grounding—which connects models to verified sources to reduce hallucinations—while noting that websites will remain visible in Bing’s search results regardless of these preferences.

While Anthropic has already integrated this signal, Google and Microsoft plan to support it starting next year, and OpenAI stated it will implement the feature in the “coming months.”