Inference Costs

A computer program that learns from examples goes through two very different phases. First, it learns for weeks from huge amounts of data, which happens only once and is very expensive. After that, the finished program is used: someone asks a question, the computer works for a few seconds, and an answer comes out. This use is called inference, and each time it costs electricity, computing time, and space in a data center anew. These exact ongoing expenses per request are the inference costs. You can think of them like the fuel costs of a car: the purchase was expensive once, but every mile driven keeps costing money.

Why every single answer shows up on the bill

Training a large model is a one-time investment. Inference, on the other hand, is a continuous burden that grows with every user. A chatbot with a hundred million users answers billions of requests every day. Even if a single answer costs only a fraction of a cent, this still adds up to enormous amounts.

That is why inference costs are a central economic metric for providers. They determine whether a subscription for 20 euros a month is profitable or a loss-making business. Heavy users can, in effect, blow up such a subscription because they consume more computing time than they pay for. Some companies therefore limit how many requests are allowed per hour.

This figure also matters for the stock market. Falling inference costs are what make new applications affordable in the first place, such as AI in every search query or every word processor. Rising costs, conversely, slow down adoption. A common misconception is that training costs above all determine success. In the long run, inference is the bigger line item.

What makes up the price per request

Billing is usually done by tokens. A token is a text unit, roughly half a syllable to a full syllable. Providers quote prices like a few cents per million tokens. In this context, the input text often costs less than the output text, because the model has to generate the answer piece by piece.

The actual cost driver is the hardware. AI models run on specialized graphics processors that cost tens of thousands of euros and draw a lot of power. The bigger the model, the more of these chips a single request needs. On top of that come cooling, networking, and the staff who operate the data center.

There are several levers for driving down costs. With quantization, the numbers inside the model are stored more coarsely, for example with 8 instead of 16 digits of precision. With Mixture of Experts, the model activates only a small portion of its building blocks per request. And with caching, the system remembers frequent requests instead of recalculating them. Together, these methods have lowered the price per token by more than a hundredfold in just a few years.

Where this figure shows up in news and products

In the quarterly reports of tech companies, inference costs appear indirectly, usually as investments in data centers and as pressure on profit margins. When a company announces that its new model is ten times cheaper to run, that is a statement about exactly these costs. Such announcements regularly move the stock prices of chipmakers and cloud providers.

As a user, you notice these costs through product decisions. Free versions of chatbots use smaller, cheaper models. For the most powerful model, you have to pay, or you may only call it a limited number of times. The distinction between fast answers and long deliberation also depends on this: a model that computes for longer costs the provider more.

One distinction helps when reading the news. Training costs are the headlines about billions spent on a single new model. Inference costs are the quiet, ongoing expenses that come afterward. Anyone who confuses the two misunderstands the economics of the AI industry.