Google’s AI advantage: why crawler separation is the only path to a fair Internet

This post argues that Google's dual use of its search crawler (Googlebot) for both indexing and generative AI training gives it an unfair competitive advantage over publishers and rival AI companies.

• •Cloudflare data shows Googlebot accesses ~1.7x more unique URLs than ClaudeBot, ~3x more than Bingbot, and ~167x more than PerplexityBot, reflecting its dominance.
• •Publishers cannot block Googlebot without losing critical search traffic and ad revenue, forcing them to implicitly consent to AI content use.
• •The UK CMA has designated Google as having Strategic Market Status (SMS) in search and search advertising, including AI Overviews and AI Mode.
• •The CMA's proposed conduct requirements would give publishers meaningful opt-out controls, require transparency documentation, and mandate effective content attribution.
• •Cloudflare argues the CMA proposals don't go far enough and that crawler separation—distinct crawlers for search vs. AI—is the only path to a fair internet.