AI news story
AI agent crawlers now need permission. Here’s how to get it
AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default…
Editor's take
Cloudflare will begin blocking AI agent crawlers by default on a significant portion of the web starting September 15th, requiring explicit opt-in for these bots to access content.
This development directly impacts how AI models, such as those powering services like Perplexity AI, access and process real-time web data. The move addresses concerns around unauthorized data scraping and the potential for these agents to overwhelm websites, a growing issue as AI-driven search and information retrieval tools become more prevalent. It signals a shift towards greater control for website owners over who can consume their content and how.
Future developments to monitor include the adoption rate of Cloudflare's proposed opt-in mechanisms by AI developers and website administrators. The effectiveness of this measure in preventing unwanted crawling, and whether it prompts alternative data acquisition strategies from AI companies, will be key indicators of its long-term impact on the AI information ecosystem.