Cloudflare has launched a tool designed to help website owners block unauthorized bot crawlers from harvesting their content without consent. Announced on July 1, this new bot blocker lets site operators decide whether AI crawlers can access their material and even set fees through a “pay per crawl” model.
For years, the internet has depended on a balance where search engines index content and direct visitors to the original sites, creating both audience engagement and ad revenue for publishers. But Cloudflare argues that this system is now threatened by AI companies scraping articles, images, and other content to power chatbots and generative tools, often without credit or compensation to the creators.
Matthew Prince, Cloudflare’s co-founder and CEO, said this shift risks eroding the financial and creative incentives that fuel quality content online. He stressed that preserving the internet requires empowering publishers with tools to control how their work is used. According to Prince, AI crawlers have been collecting data without limits, and this bot blocker aims to give creators more authority while still allowing responsible AI development.
Experts have pointed out that unrestricted scraping doesn’t just impact revenue. It can slow website performance, overload servers, and distort analytics data, driving up costs and hurting the user experience. A previous PYMNTS report noted these risks and highlighted debates about whether anti-scraping tools truly work. Some observers remain skeptical of their effectiveness, while others see promise in Cloudflare’s approach.
Cloudflare says its solution selectively blocks AI bots while allowing human visitors and search engine crawlers to access content, helping sites maintain their SEO rankings. Last year, the company introduced a similar protective feature, and industry leaders like Naxisweb’s CEO praised it for targeting AI scrapers without harming legitimate traffic.
With this latest move, Cloudflare hopes to build a fairer online environment that rewards original creators and sustains the open web amid rapid AI advances.
