On Monday, Cloudflare CEO Matthew Prince declared to launch a marketplace this coming year. It will allow website owners to charge the AI bots for scrapping their data. The marketplace empowers the publishers on how and when their website’s data should be scraped.
In an interview with TechCrunch Prince stated:
“If you don’t compensate creators one way or another, then they stop creating, and that’s the bit which has to get solved.”
In an exciting first step of its new initiative, Cloudflare unveiled AI Audit, a suite of free observability tools for customers on Monday. This powerful dashboard gives website owners insightful analytics, revealing why, when, and how often AI models are crawling their sites for information.
With AI Audit, users can take control and optimize their web presence like never before. With AI Audit, website owners have the power to block all web scrapers or selectively allow certain ones in, especially if they have partnerships or find their scraping beneficial.
This flexibility gives users greater control over their site’s data and interactions!
A demo showcasing how AI Audits work was shared with TechCrunch, highlighting tools that allow owners to track the source of each scraper. Users can see how often scrapers from OpenAI, Meta, Amazon, and other AI model providers visit their sites.
Why Cloudflare Marketplace?
Cloudflare is trying to address issues faced by the AI industry. How will smaller publishers survive if people go to ChatGPT instead of their site? It’s a really big concern as in this era AI model providers scrape thousands of small website data to empower their LLMs.
While larger publishers have secured licensing deals with OpenAI, most websites receive nothing yet see their content used in popular AI models daily. This dynamic threatens the business models of many sites, leading to a decline in the traffic they rely on.
Earlier this summer, AI-powered search startup Perplexity faced accusations of scraping websites that requested not to be crawled via the Robots Exclusion Protocol. In response, Cloudflare quickly introduced a one-click button, empowering customers to block all AI bots effortlessly.
Prince:
“That was out of frustration we were hearing, where people were feeling like their content was being stolen.”
Moreover, most website owners told business insiders that the intensity of AI bots scrapping on their site is so high that they feel like it was a DDoS attack. While scraping itself isn’t necessarily problematic, it can significantly increase cloud costs and impact service performance.
Cloudflare is addressing customer demands for control over AI bot scraping to their sites. Even large publishers with licensing deals with OpenAI—like TIME, Condé Nast, and The Atlantic—lack visibility into how much ChatGPT scrapes their content, often relying on OpenAI’s reports to gauge the value of their agreements.
To enhance transparency, Cloudflare’s upcoming marketplace will empower small publishers to negotiate deals with AI model providers.
Prince:
“Let’s give all of you the ability to do what only Reddit, Quora, and the big publishers of the world have done previously. What if we let you set, effectively, a price for accessing and taking your content to ingest into these systems.”
Although the Cloudflare marketplace is a bold move, details regarding pricing or charging bots are still unclear.
While the idea of helping small companies stand out is commendable, AI companies remain hesitant to pay for data they currently access for free. However, Cloudflare’s CEO believes this initiative is beneficial for the overall AI ecosystem.
