Cloudflare will block mixed AI crawlers from ad-supported sites starting September 15

Cloudflare is changing how AI companies can crawl the web, and the deadline is September 15, 2026. From that date, so-called mixed crawlers—bots that simultaneously scrape data for AI training, index pages for search, and power AI agents—will be automatically blocked from ad-supported sites. The move affects new customers, new sites, and all free-tier accounts unless owners opt out before the cutoff.

The separation problem

The core issue is that some of the biggest names in AI run one bot that does several jobs at once. Cloudflare's own analysis puts the stakes in sharp relief: Google's crawler generates roughly one referral visit for every 14 pages it crawls, while OpenAI's ratio sits at 1,700:1 and Anthropic's at 73,000:1, per Let's Data Science. Publishers bear the bandwidth cost; AI companies capture the value.

Cloudflare's announcement singles out Googlebot as an example of a mixed crawler—it handles both Google Search and elements of Gemini, AI Overviews, and AI Mode. Google does already operate a separate crawler called Google-Extended for AI purposes, which gives it a compliance advantage. OpenAI and Anthropic will need to ensure their crawlers are clearly separated to retain access to any site Cloudflare protects.

Cloudflare handles roughly 20% of global web traffic, which makes this enforceable at a scale no individual publisher could achieve alone.

Pay Per Use

Alongside the new defaults, Cloudflare is renaming and upgrading its Pay Per Crawl program to Pay Per Use. The original scheme charged AI companies when they fetched a page; the new model goes further, compensating publishers when their content actually surfaces in a chatbot's answer. That distinction matters: a page can be crawled thousands of times without ever generating a visible response, so linking payment to usage rather than access should close a significant gap.

Current Pay Per Use partners are Ceramic.AI and You.com. Cloudflare says the program is open to other AI companies willing to join.

What changes for publishers

Before September 15, site owners had to actively configure crawler access themselves. After that date, search indexing remains allowed by default, but AI training and agent use on monetized pages is blocked unless a site owner explicitly permits it—or an AI company pays for access via Pay Per Use.

US publishers have been fighting this battle in court, with several suing OpenAI over training data. Studies suggest AI Overviews cut organic click-through rates by around 40%, giving publishers a direct financial incentive to embrace Cloudflare's new defaults. The September deadline gives AI companies roughly two months to get their crawler strategies in order, reports TechCrunch.