For the first time since generative AI went mainstream, a major infrastructure company just changed the rules in a way that actually helps publishers instead of the companies scraping their content.
Cloudflare announced today that starting September 15, AI crawlers will be blocked by default on new customer websites and ad-supported pages. Before, you had to actively flip the switch to block them. Now you have to actively flip it to let them in.
That sounds small. It's not.
The Game Changed Last Year
When Cloudflare introduced single-click AI blocking in September 2024, they were testing something. Could publishers actually say no? Over 1 million customers answered that question by enabling the block. By now, more than 2.5 million sites have disallowed AI crawlers entirely.
That's not nothing. That's a signal. And Cloudflare just listened.
The real leverage move is the part about "mixed-use crawlers." Most AI companies run the same crawler for both search indexing (Google, Bing style) and training data (feeding Claude, GPT, etc.). Cloudflare is saying: separate those functions or get blocked from ad-supported content. Pick one job.
This forces a choice. Do you disclose what you're actually doing? Or do you lose access to half the internet?
Why This Matters for Brands
You have a problem right now. Your content is being used to train AI models, those models are answering questions that used to send traffic to your site, and nobody's compensating you for it.
Cloudflare is building what they call a "Pay Per Use" model on top of the blocking framework. Instead of paying per crawl (which they introduced last year), AI companies would pay based on how much your content actually contributes to their generated answers.
That's a revenue model for publishers. It's not a guarantee, but at least a framework that acknowledges the extraction happening.
For brands, it means leverage. If your content is valuable enough that Cloudflare customers refuse to block AI crawlers from accessing it, you can negotiate terms. You're not just hoping to rank; you're protecting your asset.
The Real Test: What Actually Happens in September
The announcement is clean. The execution gets messy.
Cloudflare is giving AI companies 75 days to separate their crawlers. Some will. Some won't and will just accept the blocks. OpenAI and Anthropic have both offered opt-out mechanisms (robots.txt overrides), but those only work if you're actively using them.
Google faces a different problem. Googlebot does double duty (search + other functions). Separating those means rewriting their infrastructure.
The September deadline isn't a cliff. It's a forcing function. By making blocking the default, Cloudflare is telling the market that this is the baseline expectation now. Publishers are not opt-in participants anymore. They're the gatekeepers.
That's the shift.
Whether it sticks depends on adoption. If enough publishers use Cloudflare's defaults, AI companies have to adapt. If most don't care and leave defaults off, nothing changes.
The bet here is that awareness matters. Showing publishers they have 2.5 million examples of sites already saying no changes behavior.
What You Should Do Now
If you're publishing content online, check your Cloudflare settings in September. The defaults will protect you by default. Decide what you actually want. Do you want to block AI crawlers entirely? Do you want to allow search but block training? Do you want to allow both but get paid?
The framework is there. You just have to choose.
That's more power than publishers had last week.