Long Tail

If you write down how often individual things are requested, almost the same pattern always emerges. At the very front stand a few things that are asked for extremely often. Behind them follows a very long row of things that are each only rarely requested. If you draw this as a curve, the back part looks like a long, flat tail. It is precisely this back area that is called the Long Tail. The astonishing thing about it: although each individual point there is tiny, the sum of all the points often exceeds the few frontrunners at the top.

Why the sum of niches tips the scales

A normal bookstore may have room for 20,000 books. It therefore mainly stocks what is sure to sell. An online retailer doesn’t have this space problem. It can offer millions of titles, even ones that are ordered only twice a year. It is precisely from these rare orders that a large part of its revenue arises.

The term was popularized in 2004 by the journalist Chris Anderson. His thesis: digital stores earn differently than classic stores. Storage costs and shelf space play almost no role anymore. This suddenly makes the niche offering worthwhile too.

For AI systems, the same pattern applies in a different form. A self-driving car handles normal roads fairly well and quickly. What’s difficult are rare situations: an overturned trailer, a person in costume, snowfall against the light. Each of these situations is unlikely on its own. Together, however, they occur constantly, and it is precisely on them that systems fail.

How the curve arises

All cases are sorted by frequency, from the most common to the rarest. They are then plotted side by side. On the left a tall, narrow peak emerges, the head of the distribution. Toward the right, the curve drops steeply and then continues very flat, often over thousands of positions. Experts speak of a power-law distribution, because the frequency keeps declining according to a fixed rule, without ever quite reaching zero.

The difference from the familiar bell curve is important. With a bell curve, such as for body heights, almost all values lie close to the average. Extremes there are genuine exceptions. With a Long Tail distribution, on the other hand, there is no meaningful average. The statement 'the typical search term' simply makes no sense here.

For AI training this has serious consequences. If you simply collect data from the web, you mainly get the head of the distribution. Rare cases are barely represented, so the model learns them poorly. That’s why companies specifically gather additional rare examples or generate them artificially. A common misconception is that more data would automatically solve the problem. Twice as much random data usually just brings more of the same.

The long tail in search, shops, and chatbots

At Google, a considerable portion of all search queries consists of phrasings that have never been entered before in that exact form. Such rare, usually longer queries are called long-tail keywords. Small websites try to rank precisely there, because competition for everyday terms is too fierce. Instead of 'running shoes,' one targets 'running shoes for wide feet in the rain.'

Streaming services operate on the same principle. Their recommendation systems are meant to lead users away from the charts and into the catalog. That’s because a subscription is more likely to be kept if someone also finds obscure titles that aren’t available anywhere else.

In news about language models, the term usually comes up in connection with errors. Models answer standard questions reliably and fail with rare specialist knowledge. Someone asking about a well-known person often gets correct information. Someone asking about an obscure local club may get pure invention. The Long Tail is thus not a marginal issue, but the actual construction site.