Cloudflare has set a new deadline for the artificial intelligence industry, requiring a clear distinction between web crawlers designed for conventional search engines, such as Google Search, and those employed for AI agents and training. As of September 15, 2026, Cloudflare's default configurations will automatically prevent "mixed-use" crawlers from accessing any web pages that display advertisements, according to the company's Wednesday announcement.
Consequently, crawlers that combine functionalities for search indexing, AI agent operations, and model training will be blocked from accessing these specific sites by default, unless the respective site owner manually modifies their settings. Cloudflare states that these default adjustments will be implemented for all new Cloudflare customers, any new websites established by current customers, and every existing free customer.
This initiative is poised to significantly influence the methods by which AI model providers acquire web content for both training their models and fueling their agentic services.
Cloudflare highlights that while the majority of website owners desire their content to be discoverable through search engines and frequently via AI services, they simultaneously seek robust protections against their intellectual property being utilized without compensation.
Cloudflare explicitly singled out the "world’s largest search engine," an evident reference to Google, asserting that it possesses access to approximately "2x more information" than other AI entities. This advantage, Cloudflare suggests, stems from the search giant making it challenging for customers to ensure their content's discoverability without it simultaneously being leveraged for AI purposes.
Google has previously countered such generalizations, noting its provision of a bot named Google Extended, which allows site owners to opt out of having their content used for AI training and various AI products and services, including Gemini Apps and Vertex API. Importantly, utilizing Google Extended does not affect a site's inclusion in Google Search. Nevertheless, the tech giant's primary Googlebot is responsible for crawling content for Search, which encompasses AI-driven features such as AI Overviews and AI Mode.
"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," stated Matthew Prince, Cloudflare co-founder and CEO, during his announcement. He was referring to the recent significant milestone where, for the first time, bots surpassed human traffic online—a shift that was not anticipated until the following year.
Prince further elaborated, saying, "Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training."
While Cloudflare provides various products designed to assist users in deploying their own AI systems, the company has concurrently introduced an array of tools empowering publishers with greater control over their content within the evolving AI landscape. In recent years, Cloudflare has notably launched initiatives to counter AI bots, including a dedicated marketplace, termed "Pay Per Crawl," which enables websites to levy charges against AI bots for scraping their content.
This "Pay Per Crawl" model is now evolving into "Pay Per Use," as announced by the company. This progression will permit publishers to bill AI companies based on the value generated by their content, rather than merely when the content is retrieved.
Furthermore, this modification holds the potential to conserve significant bandwidth and compute resources for publishers, especially concerning AI model providers. Cloudflare's internal data indicates that over 50% of crawl traffic originating from AI crawlers is currently expended on re-fetching web pages that have not undergone any changes.
To implement these new capabilities, Cloudflare is commencing its efforts with two initial partners: Ceramic.ai and You.com. Publishers who opt into this system will receive payment when their content is featured in Ceramic’s AI search results or when You.com accesses their premium content.
Cloudflare notes that other AI companies will have the flexibility to customize this model to align with their specific operational frameworks.
The Editorial Staff at AIChief is a team of professional content writers with extensive experience in AI and marketing. Founded in 2025, AIChief has quickly grown into the largest free AI resource hub in the industry.