Scaling Laws

AI systems learn from examples: a computer program is shown a great deal of text, and from this it learns which words typically follow one another. This raises a practical question: what happens if you make the program larger, show it more text, or let it compute for longer? Scaling laws are the answer to this. They are formulas derived from series of measurements that describe how strongly the error rate falls when these three quantities are increased. The surprising finding: the improvement follows a very smooth curve, across many orders of magnitude. So you can measure at small scale and predict at large scale.

Why this turns into billion-dollar budgets

Without scaling laws, building a large AI model would be a gamble. A training run costs tens of millions of dollars depending on size and takes weeks. No one spends that money without an estimate of the outcome. It is precisely this estimate that the laws provide.

In practice, the process looks like this: a lab trains several small models with varying amounts of effort. From the measured data points it draws a curve and extends it. The curve then indicates, for example, how good a model will be that consumes a hundred times as much compute. Only once this forecast is convincing is the large run started.

Scaling laws thus also explain a development of recent years. There was no need to find a fundamentally new idea in order to build better language models. For a long time, it was enough to repeat the same recipe with more compute. This is the actual reason for the enormous demand for data centers and specialized chips.

The three levers and their ratio

Three quantities matter. First, the number of parameters, i.e., the internal adjustable values that the model tunes during learning. Second, the amount of training data. Third, the compute, usually measured in total computational operations. If any one of these quantities grows, the model’s prediction error falls.

The ratio matters. A widely noted 2022 study, known after the Chinchilla model, showed that many models were too large for their amount of data. Given a fixed compute budget, it pays off to increase parameters and data roughly in lockstep. As a rule of thumb, the researchers cited around twenty training words per parameter. Earlier models had in some cases seen ten times less data per parameter and thereby gave up performance.

A common misconception is that the curves continue indefinitely. They do not, without limit. The relationship is a power law: each further improvement costs a multiple of the previous effort. Ten times more compute does not halve the error, but only brings a small step forward. Moreover, the laws only say something about the measurable error rate, not about whether a model truly masters a given task.

Scaling laws in news and products

When business news reports that a corporation is investing tens of billions of dollars in data centers, this logic lies behind it. Statements like “the next model generation will use ten times as much compute” also refer to this. The share price of chipmaker Nvidia is indirectly tied to the belief that the curves will continue to hold.

At the same time, there has been discussion for a while now about whether a limit is drawing nearer. High-quality text on the internet is finite, and data is one of the three levers. That is why labs are working on alternatives: artificially generated training data, or more compute not during training but when answering a question. The latter is called inference-time scaling, and models that “think” longer before answering make use of exactly this.

For users this mostly remains invisible. It is noticeable indirectly: smaller models that run on a phone are today as good as large models from two years ago. This too is a consequence of the insight that a smart distribution of data and size achieves more than sheer size alone.