Crawl-to-Refer Ratio
The crawl-to-refer ratio compares how often a search engine's or AI provider's program automatically reads a website with the number of visitors that provider sends back in return. It has become the central metric in the debate over whether AI providers treat publishers fairly.
Large providers like Google or OpenAI run programs across the internet that automatically read and store web pages. Such programs are called crawlers. Their haul ends up in search engines or serves as training material for AI systems. Conversely, the same providers send people back to the websites, for instance when someone clicks on a search result. These returned visitors are called referrals. The crawl-to-refer ratio sets both figures against each other: how many page requests by the crawler correspond to a single real visitor?
Why publishers argue over this figure
For websites that live off advertising or subscriptions, visitor traffic is hard cash. For decades, an unspoken deal applied: the search engine may read and index, and in return it delivers readers. Both sides benefited. The crawl-to-refer ratio now measures whether this deal still holds up.
The published figures show clear differences between providers. For classic Google search, it long amounted to roughly a few dozen requests per visitor. For AI providers' crawlers, however, the infrastructure company Cloudflare reported figures in 2025 ranging from several hundred to over a thousand requests per visitor. The reason is obvious: a chatbot summarizes the answer itself. Users then click through to the source less often.
A high ratio therefore means: the website supplies a lot of content and gets little back in return. Publishers use this metric as an argument in negotiations over licensing fees. Some also use it to decide which crawlers they still allow onto their servers at all.
How the figure is calculated
Every server keeps a log of all accesses. Each request includes an identifier known as the user agent. Crawlers identify themselves there with a fixed name, such as GPTBot or Googlebot. One simply counts the lines bearing this name.
Visitors are counted via the referrer. This is the entry in the log indicating which page a user came from. If it shows a ChatGPT address, it was a referral from OpenAI. Dividing the two figures yields the ratio, usually written as “X:1”.
The metric is, however, cruder than it appears. Some crawlers collect training data, others merely fetch a single page for a current query. Both flow into the same number. Moreover, some programs disguise themselves as ordinary browsers and don’t show up at all. And anyone who reads an answer in the chatbot without clicking through has still perceived the brand. This value cannot be measured.
Where the metric turns up
The figure appears most often in news coverage of the conflict between media companies and AI firms. Cloudflare, which handles traffic for a large share of all websites, regularly publishes corresponding analyses. These figures then resurface in reports on lawsuits and licensing deals.
In practice, this also has technical consequences. Websites control which crawlers they allow via a file called robots.txt. Anyone who notices a very unfavorable ratio blocks individual AI crawlers there. Since 2025, Cloudflare has additionally offered payment models in which crawlers pay per request.
It is important to distinguish this from similar terms. Click-through rate measures how many people click on a displayed result. “Zero-click search” describes search queries with no click-through at all. The crawl-to-refer ratio, by contrast, expressly considers the exchange: computing load and content on one side, visitors on the other.