Cloudflare has unveiled a new policy designed to give publishers more control over how AI systems access online content. Beginning on September 15, 2026, the company's default settings will block mixed-use crawlers from pages that run ads, unless site owners choose otherwise.
The change separates crawlers used for traditional search from those that also support AI training and agent-based services. It will apply to new Cloudflare customers, newly created sites for existing customers, and all current free accounts by default.
According to Cloudflare, the update reflects a broader shift in the web economy, where publishers want visibility in search while also protecting the value of their work. The company says many site owners are open to AI discovery, but want clearer rules around how their content is used.
Cloudflare also noted that major search platforms often have broader access to web data than smaller AI companies. In response, some providers have already introduced opt-out tools for training and AI features, while keeping search indexing separate.
Alongside the policy change, Cloudflare is expanding its monetization tools for publishers. Its earlier Pay Per Crawl approach is evolving into Pay Per Use, a model that would let publishers earn revenue when their content creates value for AI systems, not only when it is retrieved.
The company says this could also reduce repeated crawling, since a large share of AI bot traffic is spent revisiting pages that have not changed. Cloudflare is launching the model with Ceramic.ai and You.com, while allowing other AI companies to adapt the framework to their own systems.
This shift points toward a future where content access, attribution, and compensation may become more closely connected across the digital ecosystem.
