Inference

A computer program that learns from examples goes through two very different phases. In the first phase, training, it is presented with huge amounts of sample texts or images. In doing so, it adjusts millions of internal parameters so that it can reproduce typical patterns. This phase can take weeks and happens only once. In the second phase, inference, the learning is complete. You enter a question, the program computes with it, and delivers an answer. This takes seconds, but happens anew with every single use.

Why inference writes the real bill

Training gets a lot of attention because the numbers involved are spectacular. Training a large language model can cost hundreds of millions of dollars. But that is a one-time investment. Inference, on the other hand, runs daily and without end.

Work through an example. A chatbot with a hundred million users easily answers a billion requests per month. Even if each answer costs only a fraction of a cent, this still adds up to millions of dollars. That’s why whether an AI product makes money usually hinges on inference costs, not on training.

This also explains why chipmakers like Nvidia benefit so strongly from the AI boom. Their graphics processing units, GPUs for short, are specialized precisely for this kind of computation. A growing share of the AI chips sold worldwide now no longer work in training, but run around the clock in inference.

What happens inside the model when it answers

A language model does not write finished sentences all at once. It only ever predicts the next piece of a word. It then appends this piece to the input and computes again. This is how an answer emerges piece by piece, which you can see in the typing-like appearance of the text.

Each of these steps involves a very large number of multiplications with the model’s stored parameters. Because nothing is being learned anymore, these values remain unchanged throughout. This is precisely why inference is much faster than training. It still requires an enormous amount of memory, though, since the entire model must be kept ready at hand in the chip.

To make this cheaper, there are two common tricks. The first, called quantization, stores the numbers in the model more coarsely, for example with eight instead of sixteen decimal places. The second, Mixture of Experts, has the model activate only a small portion of its building blocks per request. Both significantly reduce costs without noticeably harming quality.

Inference on your phone and in quarterly earnings

Every time you ask ChatGPT something, start a translation, or your phone automatically brightens a photo, inference is running. Most of the time this happens in a data center, and your input travels across the internet for it. Increasingly, however, models compute directly on the device. This is called on-device inference, and it has the advantage that no data leaves the phone.

In business news, the term mainly comes up in two contexts. First, when companies explain their data center spending. Second, when a new model advertises that it needs less computing time per answer at the same quality. The second case is often economically more important than a few extra percentage points of accuracy.

A common misconception is worth addressing in closing: many assume a model learns from every conversation. During inference, exactly that does not happen. The model remains unchanged and forgets the conversation afterward. Whatever it learns only enters through a later, separate training round.