Inference Cost

An AI program must first learn from vast amounts of data. This learning phase is called training and runs once, often over weeks. After that, everyday operation begins: users ask questions, and the finished program computes an answer. This computational work during ongoing operation is called inference. Inference costs are what each of these individual answers costs in electricity, computing time, and hardware. They don’t arise once, but freshly with every single request.

Why they determine profit or loss

Training a large model costs a lot of money all at once. After that, this bill is paid. Inference costs, on the other hand, grow with every new user. A service with a hundred million users pays a hundred million times. That’s why a successful product can push its provider into the red.

This explains a pattern often seen in business news. AI companies limit the number of queries per day or charge subscription fees. They switch to smaller, cheaper models for simple questions. And they buy enormous quantities of graphics cards just to have enough computing capacity at all. All of this is a reaction to ongoing inference costs.

A common misconception is that AI is expensive primarily because of training. Over the lifetime of a popular product, inference costs usually far exceed training costs. Analysts estimate that for heavily used services, the majority of data center capacity flows into inference, not training.

What drives up the price per answer

The most important factor is the size of the model. Models consist of parameters, which are adjustable numbers determined during training. The more parameters, the more computational steps per answer. A model with many hundreds of billions of parameters needs several expensive specialized chips running simultaneously just to operate at all.

The second factor is the length of the text. Models break language down into small building blocks called tokens, roughly parts of words. Billing is usually done per thousand tokens, for both the question and the answer. Anyone who copies an entire PDF into the chat therefore incurs significantly higher costs than with a short question. Models that “think” extensively before answering and generate a lot of intermediate text are expensive for the same reason.

There are technical countermeasures against both. With quantization, the numbers in the model are stored more coarsely, for example with eight instead of sixteen decimal bits. With distillation, a small model is trained to mimic a large one. And with the mixture-of-experts architecture, the model activates only a fraction of its building blocks per query. Such techniques have reduced the cost per million tokens by orders of magnitude in recent years.

Where you encounter the bill in everyday life

This is most visible in providers' price lists. OpenAI, Google, and Anthropic list prices per million tokens, separately for input and output. Anyone building an app that uses such a model pays exactly these amounts. That’s why developers deliberately choose smaller models for simple tasks.

As a regular user, you also feel the costs indirectly. Free versions of chatbots use weaker models or have message limits. Paid versions grant access to the large models. Some features now run directly on the phone, because that incurs no server costs for the provider.

In stock market news, the term comes up in the context of data centers and power consumption. Rising inference costs are one reason why tech companies are pouring billions into chips and their own power plants. An important distinction here: inference costs are purely operating costs and are not the same as the price you ultimately pay. How much of that the customer bears is determined by the business model.