The new Cloudflare AI crawler rules draw a hard line through the modern web. Cloudflare laid out the plan in a press release published Wednesday. Bots that blur search and AI harvesting will soon hit a default block. Moreover, a new Pay Per Use program will pay publishers when their content actually powers AI answers.
The philosophy behind the package is blunt: your content, your rules. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said co-founder and CEO Matthew Prince. In other words, Cloudflare wants bots to identify themselves, state their purpose, and pay their way.
Mixed Crawlers Lose Their Free Pass on September 15
The sharpest change targets so-called mixed crawlers. These bots collect content for search and AI training in a single pass. As a result, site owners cannot allow one use without feeding the other. Starting September 15, Cloudflare will block mixed crawlers that refuse to separate search, agent use, and training. The block covers every page that carries ads.
Defaults are shifting at the same time. New customers will allow search crawling but block AI training and agent use on ad pages. New sites from existing customers get the same treatment. Meanwhile, free-plan customers who change nothing before the deadline will inherit those defaults. Every customer can still override the settings from the dashboard.
Cloudflare never names Google, but the subtext is hard to miss. The company notes that “the largest search engine accesses roughly two times more information than leading AI companies.” Consequently, publishers face an ugly choice today. They can block a mixed crawler and vanish from search, or stay visible and feed AI products for free.
Pay Per Use Pays for Answers, Not Fetches
The compensation side also grew up. Pay Per Use evolves last year’s Pay Per Crawl experiment. It pays publishers when content creates value, not merely when a bot fetches a page. Ceramic.ai is first in line with a pay-per-query model that returns queries, citations, and ranking data to publishers. You.com joins with on-demand payments for premium content.
Beyond that, beehiiv is giving newsletter creators dashboards to track, block, or opt into AI models. Patreon, meanwhile, blocks known training crawlers and still allows discovery. The direction mirrors the rest of the industry. Content owners increasingly want paid, attributed pipelines. The same pressure produced the Getty Images licensing deal for ChatGPT Search and DeepMind’s research partnership with A24. In fact, Cloudflare says content owners signed more than 50 major licensing agreements in the past year alone.
New Plumbing for an Agentic Web
Several supporting tools round out the announcement. An Attribution Business Insights dashboard shows which AI bots consume a site’s content. It also reveals how much human traffic each AI company sends back in return. Additionally, an Answer Engine Optimization layer helps sites measure how they surface inside AI answers instead of classic search results.
Efficiency gets attention too. According to Cloudflare, over half of AI crawl traffic re-fetches pages that have not changed. New content signals will tell bots when a page is stale. Meanwhile, a Web Bot Auth framework lets well-behaved crawlers verify their identity. Together, the pieces sketch a web that welcomes agents, but only on the record.
What It Means for the Open Web
The details could still change. Cloudflare describes the next two months as an engagement and testing window, so the September defaults could still bend. Furthermore, the plan only works if AI companies actually adopt verified identities and paid access instead of routing around them. The biggest question mark remains Google, which has shown no public appetite for splitting its crawler.
Still, the direction is clear. Cloudflare sits in front of a huge share of the web. Its defaults therefore become de facto policy for millions of sites. For now, publishers get more visibility, more leverage, and a new way to get paid. AI crawlers get a deadline.
