Cloudflare can't block Google's AI scraper without killing publisher search traffic
Jul 9, 2025
Key Points
- Google's AI scraper and search indexer share the same user agent, making it technically impossible for Cloudflare's filter to block one without blocking the other.
- Publishers face an impossible choice: protect content from AI training or preserve Google Search traffic, which most depend on for discovery.
- Cloudflare CEO Matthew Prince says the tension exposes a fundamental problem with AI regulation tools that lack the granularity to distinguish between search and data extraction.
Summary
Cloudflare's default AI bot filter cannot block Google's Gemini scraper without destroying publisher search traffic. Google's AI scraper and Google's search indexer use the same user agent, so blocking one means blocking the other. For most publishers, Google organic search is a critical traffic source, and the filter has no way to distinguish between the two because they operate under the same identity.
This creates a bind for publishers who want to protect their content from AI training but depend on Google for discovery. Matthew Prince, Cloudflare's CEO, has highlighted this as a core tension in AI regulation and content protection. The tools designed to block AI scrapers often lack the granularity needed to separate beneficial search traffic from data extraction. Publishers face a choice between losing AI protection or losing search visibility, and technical solutions alone cannot resolve the underlying conflict.