Cloudflare CEO says the web's future is "pay to crawl" as bots overtake human traffic

Editor's take

Bot traffic has surpassed human visits to websites, a milestone that arrived years earlier than anticipated, driven by the proliferation of AI agents. This shift fundamentally alters the economics of the internet, as search engines and AI training models consume vast bandwidth and resources without direct compensation to content creators. The implication is a potential restructuring of how information is accessed and monetized online.

This development directly impacts content publishers, search engine providers like Google, and AI developers such as OpenAI. If AI agents are the primary drivers of internet traffic, then the existing advertising-based revenue model for many websites becomes unsustainable. The "pay to crawl" model, while conceptually simple, raises significant questions about accessibility, censorship, and the very nature of an open internet.

The next crucial development will be how this "pay to crawl" concept is implemented. Will it be a universal fee, or tiered access based on usage? What mechanisms will be put in place to verify bot versus human traffic, and how will this affect smaller publishers or researchers? The evolution of this model will determine whether the internet becomes a more exclusive, subscription-based information ecosystem or finds alternative ways to balance resource consumption with open access.