Cloudflare CEO Matthew Prince is urging the UK’s Competition and Markets Authority (CMA) to introduce new rules that would separate Google’s AI crawlers from its search crawlers, arguing that the tech giant’s dual use of its web-scraping technology gives it an unfair advantage in the AI race.
Speaking at the Bloomberg Tech conference in London, Prince said Cloudflare — which serves as an internet infrastructure backbone for most of the web — is uniquely positioned to comment on this issue.
“We don’t have a dog directly in the fight. We’re not an AI company or a media publisher. But 80% of AI companies are our customers,” Prince stated.
The Core of the Debate: Google’s Bundled Crawlers
At the center of the debate is Google’s long-standing web crawler, which gathers website content for both its search engine and its AI models.
According to Prince, this setup allows Google to use the same crawler to fuel AI products like Gemini and AI Overviews, while competitors such as OpenAI, Anthropic, and Perplexity must negotiate or pay for similar access to web data.
“Google is saying they have a God-given right to all of the world’s content, even if they don’t pay for it,” Prince explained. “If you want to opt out of AI training, you have to opt out of search too — and that’s simply not viable for most publishers.”
This forced choice puts media outlets and small businesses in a difficult position. Blocking Google’s crawler not only impacts search visibility — often responsible for 20% or more of online revenue — but also affects ad safety systems that regulate ad placements across Google platforms.
Cloudflare’s Push for Fair AI Competition
Prince’s comments follow Cloudflare’s recent efforts to help publishers monetize AI scraping. Earlier this year, the company launched a marketplace for websites to charge AI bots that use their data to train large language models (LLMs).
The move positioned Cloudflare as a middleman between content creators and AI developers — promoting what Prince calls a “fair data economy” where companies pay for the content they train on.
Now, he’s taking that fight to regulators. Prince told the CMA that Google’s bundled crawling practice effectively locks competitors out of the AI ecosystem, granting Google a data advantage that could “hand the game” to them.
“We could have a market where thousands of AI companies buy content from thousands of publishers — but that won’t happen unless Google is required to compete fairly,” he said.
UK CMA’s Increasing Scrutiny of Google
Earlier this month, the UK’s CMA officially designated Google as having a “substantial and entrenched position” in the search and advertising markets. The designation gives the regulator new powers to impose stricter obligations on Google — including in areas such as AI Overviews, Discover feed, Top Stories, and News tabs.
Cloudflare has reportedly submitted data to the CMA demonstrating how Google’s crawler operates and why it’s nearly impossible for others to replicate its efficiency or coverage.
Industry Support for Cloudflare’s Position
Cloudflare isn’t alone in its concerns. Neil Vogel, CEO of Dotdash Meredith, the largest U.S. digital and print publisher, also criticized Google’s crawler system last month, calling it “anti-competitive” and “unfair to media companies.”
Vogel’s company has adopted Cloudflare’s technology to block non-paying AI bots, and he claims it’s already prompting deal discussions with major LLM providers — showing early signs that AI companies are willing to pay for data access when fair systems exist.
The Bigger Picture
As AI continues to dominate the tech landscape, the battle over who controls access to web data is becoming one of the defining regulatory fights of the decade.
Prince’s message to UK regulators is clear: if AI is to remain an open, competitive ecosystem, no single company — not even Google — should be allowed to control the gateway to the world’s information.