Every publisher faces the same impossible choice right now. Block AI crawlers and watch your traffic collapse. Let them crawl and give away your content for free. There is no third option. But there is a smarter way to play it.
The Default Just Changed
On September 15, Cloudflare will make AI crawler blocking the default setting for every new customer and every newly added website. The company is specifically targeting "mixed-use" crawlers, bots that claim to index for search but quietly feed training data to AI models.
Existing free customers get the same treatment automatically unless they opt out. This is not a niche change. Cloudflare powers roughly 20% of the web.
I covered the announcement itself earlier this week. The blocking mechanism is straightforward. What nobody is talking about is the trap it creates for every brand that publishes content online.
The 23% Problem
When major publishers started blocking AI crawlers, the results were immediate and brutal. A Hacks/Hackers analysis found that among the top 30 publishers who blocked AI bots, total traffic declined by 23%. Not AI referral traffic. Total traffic.
Why? Because AI search engines like ChatGPT, Perplexity, and Google's AI Overviews are rapidly becoming consumers' primary discovery surface. When your content is not in those answers, you are invisible to the fastest-growing search channel on the internet.
I wrote about this AI visibility crisis in June. The window to get cited by AI search engines was already closing. Now Cloudflare is bolting it shut by default.
The stat that should worry every brand: 79% of news publishers already block at least one AI training bot. But only 46% block Google-Extended. Publishers are picking and choosing. They are scared of Google's AI Overview traffic disappearing, but they are not sure what the right answer is.
A Whole New Industry Just Appeared
In the past two weeks, three separate companies launched products designed to solve this exact problem from the opposite direction. They do not help you block AI crawlers. They help you get crawled more effectively.
Alli AI released a WordPress plugin that uses server-side rendering to deliver pre-rendered HTML to AI crawlers like ChatGPT, Perplexity, and Claude. The plugin ensures your content can be parsed by AI systems without changing the human user experience. More than 40% of the web runs on WordPress. This is not a side project.
CiteLens launched a Generative Engine Optimization intelligence platform that tracks brand visibility across AI search engines, runs queries across multiple AI tools, identifies cited sources, and computes a visibility score against competitors.
GegoSoft started a website design service focused entirely on AI search readiness, incorporating structured data markup and schema libraries to ensure business sites get cited by automated text models.
These are funded companies building for a market that did not exist 18 months ago. The AI visibility industry is real, and it is growing fast because the demand is desperate.
Most Sites Are Accidentally Invisible
Here is the part nobody talks about. A recent audit of thousands of US and UK websites found that 27% are accidentally blocking AI crawlers they would want to be visible to. The blocks came from outdated robots.txt rules, security plugins over-correcting, and default hosting configurations that treat all non-Google bots as hostile.
These sites are invisible to AI search engines. Their owners do not know it. Their analytics cannot show it because AI-driven traffic often registers as direct or uncategorized.
The irony is sharp. Brands are spending more on content than ever while accidentally making that content unreadable to the platforms consumers actually use to find information. They are paying to disappear.
Pay Per Use Changes the Math
Cloudflare's most interesting move is not the blocking. It is the compensation model called Pay Per Use. The idea: when an AI answer uses your content, you get paid. Not just when your site is crawled. When the content is actually referenced in an AI-generated response.
Pay Per Crawl, the precursor launched last year, let publishers charge per request. Pay Per Use goes further. Publishers set a price. AI companies that want to cite the content pay it. If they do not pay, they do not get to use the content in answers.
This is the first real attempt to build an economic model around AI content usage. It could collapse under its own weight. Or it could become the foundation of how content is monetized for the next decade.
For brands watching their attribution collapse in real time, this matters. If Google and OpenAI will not send you traffic, maybe they will send you a check instead.
The Smart Play
Most brands will pick a side. Block everything or allow everything. Both positions are wrong.
The brands that win will treat AI crawler strategy the same way they treat SEO: as an ongoing optimization, not a one-time decision. They will audit their robots.txt quarterly. They will use tools like CiteLens to track AI visibility. They will experiment with Pay Per Use pricing to see if AI companies actually pay.
The default is changing on September 15. But the default will not save you. Neither will panic-blocking every bot. The only move that works is treating AI visibility as a core marketing function, not an IT afterthought.
Cloudflare gave publishers the power to choose. Most will choose wrong.