Scaling Law

A computer program that learns from examples gets better when you show it more examples. That sounds trivial. What’s surprising is how predictable this relationship is. Researchers have built many such programs in various sizes and measured their error rate. What they found: if you multiply the effort by ten, the error drops by a fairly fixed amount — almost like a law of nature. This measurable regularity is exactly what’s called a scaling law.

Why companies are betting billions on it

A data center for AI costs billions. Nobody spends that much money without knowing what will come out of it. Scaling laws provide exactly this kind of prediction. You train a few small, cheap models, measure their error rate, and plot the points on a chart. The line through these points can be extended. This way you can estimate how good a model a hundred times more expensive will turn out to be, before spending the first dollar on it.

This predictability is what triggered the boom of recent years in the first place. Before 2020, it was unclear whether larger models would really, reliably get better. Since then, size has been considered the most reliable lever there is. That’s why corporations like Microsoft, Google, or Meta buy hundreds of thousands of specialized chips. They’re not buying hope — they’re buying a position on a known curve.

The catch lies in the shape of this curve. It doesn’t rise in a straight line, but keeps getting flatter. The first improvements are cheap, the later ones extremely expensive. To achieve the same amount of progress again, you often have to multiply the effort by ten. This is exactly what the industry is currently arguing about fiercely: is the next tenfold increase still worth it?

The three levers: size, data, compute

Three things can be scaled up. First, the number of parameters — these are the adjustable numerical values in the model, comparable to the knobs on a giant mixing console. Second, the amount of training text, measured in words or word fragments. Third, the sheer amount of compute time, i.e. how long the chips work. These three are interconnected, since compute time is roughly the product of model size and data volume.

For a long time, the main approach was to build ever-larger models and feed them comparatively little text. A 2022 study by DeepMind showed that this was a mistake. Given a fixed budget, it pays off to increase model size and data volume together. As a rough rule of thumb, the researchers cited around twenty words of training text per parameter. Their model Chinchilla was considerably smaller than the competition and yet performed better.

It’s important to note what a scaling law does not say. It predicts an error rate, i.e. how accurately the model guesses the next word. It does not predict whether the model will suddenly be able to solve equations or understand jokes. Such abilities sometimes appear quite abruptly once a certain size is exceeded. The smooth curve on the chart and the seemingly sudden leap experienced when using the model are two different things.

Why the curve is currently wobbling

Anyone reading news about AI regularly comes across the question of whether scaling laws still hold. The reason is simple: the usable text material on the internet is finite. You can build data centers, but you can’t build a second internet. Companies are therefore turning to videos, program code, or artificially generated text instead. Whether that works just as well remains an open question.

At the same time, a second curve has come into focus. Instead of scaling up training, the finished model is made to think longer at each request. Models like OpenAI’s o-series generate a long internal chain of thought before answering. Here too there’s a measurable regularity: more thinking time, fewer errors. This is referred to as scaling laws for inference rather than for training.

For the stock market, this is no academic question. The share price of chipmakers like Nvidia hinges on whether further scaling is still worthwhile. When a report includes the sentence that scaling laws are hitting their limits, it’s exactly this flattening curve that’s being talked about. And the question of where the next dollar is best invested.