Inference Economics

When you ask an AI chatbot something, a computer in a data center is crunching numbers on the other end. This computation costs electricity, hardware, and space. Experts call the process by which a finished AI model generates a response inference. Inference economics is the question of what such a response costs and how much it brings in. Unlike training the model, which happens only once, these costs arise anew with every single request. That is why inference economics determines whether an AI product makes money or burns money.

Why every response ends up on the bill

Classic software has a convenient property: once it’s written, the millionth user costs almost nothing extra. With AI, it’s different. Every request occupies an expensive specialized chip for seconds at a time. At a billion requests per day, even fractions of a cent add up to enormous sums.

This is exactly where the business problem of many AI providers arises. They sell subscriptions at a fixed price, say 20 euros a month. A user who uses the service intensively can generate more compute costs than they pay. Several providers have publicly admitted that their most expensive tiers run at a loss.

Conversely: if the cost per response falls, new products suddenly become possible. An AI that presorts every email in the background would be unaffordable at high costs. At low costs, it’s a standard feature. So inference economics determines not only profits, but also which applications exist in the first place.

The math behind a single response

Billing is usually done in tokens. A token is a text building block, such as a short word or a syllable. Providers quote prices per million tokens, separately for input and output. A German sentence consists of roughly 15 to 25 tokens. This makes it possible to calculate what a conversation costs.

Three factors drive the price. First, model size: larger models require more computational steps per token. Second, the length of the conversation, since the model re-reads the entire prior history with every response. Third, the question of how much the model thinks internally before answering. So-called reasoning models generate hundreds of invisible intermediate steps first. This improves quality and multiplies the costs.

Savings are made at all three points. Small, specialized models handle simple requests, while large ones handle only the difficult ones. Models that activate only part of their building blocks per request compute faster at equal capability. Numbers within the model are stored more coarsely, which saves computation time. And frequently asked questions are cached, so the same answer isn’t computed twice. As a result, the price per million tokens has fallen by a factor of a hundred within a few years.

Where these numbers show up in the news

In quarterly reports from Microsoft, Google, or Amazon, inference economics is the reason behind massive investment sums. The corporations are building data centers because they expect strongly rising demand for inference. Analysts then ask whether revenues from AI services will ever cover these expenditures. That is precisely the core question of inference economics.

You notice the math in everyday life too. Free versions of chatbots use smaller models or have daily limits. Paid versions give you access to the more expensive, more thorough models. This tiering is not arbitrary but a direct consequence of the different compute costs.

A common misconception is to confuse inference economics with training costs. Training a top-tier model costs hundreds of millions once and makes headlines. But over the entire lifetime of a popular model, inference costs are significantly higher. For providers, the inconspicuous ongoing burden is economically more important than the spectacular one-time investment.