Vetted by NeuralPress's Multi-Agent Verifier for strict factual validity and event relevance. Our compliance engine cross-checks and filters search results to ensure zero false correlations or misleading content.
Crawl-to-Refer Ratio by AI Provider
Comparison of how many times AI bots crawl a site versus how many referrals they send.
Primary Sources
AI Bots Are Strip-Mining the Web, and Anthropic Leads the Pack ...
Anthropic CEO Dario Amodei. Chris Ratcliffe/Bloomberg via Getty Images 2026-04-12T09:00:01.237Z Cloudflare's data shows that AI companies consume more web value than they return. Anthropic's crawl-to-refer ratio is 8,800 to 1, highlighting ethical concerns. AI chatbots reduce web traffic, upending the web's grand bargain. Cloudflare's latest data offers one of the clearest snapshots yet of how AI companies consume the web, and how little they give back. The company, which powers roughly 20% of the internet, tracks how AI bots crawl websites versus how often those platforms send users back through referrals. The resulting "crawl-to-refer" ratio is a simple yet telling metric: how much value is extracted compared to returned.The early April 2026 figures are stark. Anthropic is the worst by a wide margin, with a ratio of 8,800 to 1. That means its bots crawl webpages 8,800 times for every referral sent.OpenAI follows at 993 to 1. Microsoft, Google, and DuckDuckGo look far more balanced by comparison.Anthropic's position is particularly striking given its reputation for being "ethical." That reputation has made it a preferred choice among some users who want to support more responsible AI development. This data highlights a different dimension of ethics — how companies interact with the broader web ecosystem that provides information for AI model outputs. Historically, the internet operated on an implicit bargain: websites allowed search engines to crawl and index their content for free, and in return received traffic they could monetize. Generative AI breaks that bargain. Chatbots increasingly provide direct answers, reducing the need for users to click through to original sources.This results in a system that extracts more value than it gives back — and in some cases, increases costs for site owners due to heavy bot activity.Anthropic has previously questioned Cloudflare's methodology and pointed to growing referral traffic from new features. Still, the broader trend is hard to ignore. I asked Anthropic for comment this time, too, and it did not respond.If the web's economic engine depends on traffic and referrals, these ratios raise a fundamental question: What will incentivize the sharing of verified information online in the future?Cloudflare is trying with a new marketplace for web content. It's unclear if efforts like this will succeed. After all, what's better than using people's content for free?Sign up for BI's Tech Memo newsletter here. Reach ...
Anthropic says its most powerful AI cyber model is too dangerous to ...
Anthropic launched Project Glasswing, a $100 million AI cybersecurity initiative using its unreleased Claude Mythos Preview model to find and patch zero-day vulnerabilities across critical ...
Anthropic's Mythos Preview and the End of a Twenty-Year Cybersecurity ...
8 Apr 2026 - Anthropic yesterday announced Claude Mythos Preview, a frontier AI model that can autonomously discover and exploit zero-day vulnerabilities in every major operating system and every major web browser. Alongside the model, Anthropic launched Project Glasswing, a defensive coalition including Amazon Web Services, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the ...
Why Anthropic's new model has cybersecurity experts rattled
AI Why Anthropic's new model has cybersecurity experts rattled The company says it has built its most dangerous model yet. Can its coalition of internet companies fix the internet before others catch up?
