Token Margin

Providers of AI services don’t bill by word, but by small chunks of text. Such a chunk is called a token and usually comprises a short word or a syllable. For a thousand tokens, a provider charges a fixed price, often just fractions of a cent. At the same time, every token costs the provider money, because specialized chips in data centers work on it and consume electricity. The difference between the price and these costs is called the token margin. It is positive if the provider earns money on the sale, and negative if it loses money.

Why investors watch the cent per token

A single token costs ridiculously little. That is exactly where the trap lies. Large chatbots process billions of tokens every day, and this sheer volume turns every tiny difference into a huge sum. A tenth of a cent loss per thousand tokens sounds harmless, but at a trillion tokens per month it adds up to a nine-figure amount.

That is why the token margin matters more to analysts than revenue alone. An AI company can multiply its revenue and still slide deeper into the red every year. If a business grows with a negative margin, the loss grows along with it. It is different with classic software: there, the millionth copy of a program costs almost nothing extra.

A common misconception is that the expensive training runs of the models are the real cost problem. Training is a one-time expense that can be spread out over years. The costs of ongoing use, on the other hand, recur with every single response. Only these ongoing costs are reflected in the token margin.

What drives the calculation up and down

On the revenue side stands a simple list price. Providers typically charge less for incoming tokens, i.e. the user’s question, than for outgoing tokens, i.e. the answer. The reason lies in the technology: the system can process the question in one go, whereas the answer is generated token by token in sequence. That is considerably more resource-intensive.

On the cost side, what matters most is how long a specialized chip is occupied processing a token. Renting such chips costs several dollars per hour. The more requests a provider can pack onto a chip at the same time, the cheaper each individual token becomes. But this bundling only works if enough users are making requests simultaneously. A half-empty data center has terrible margins, similar to an airplane with twenty passengers.

Providers also push down costs through smaller models, coarser storage of the internal numerical values, and by having only part of the model compute per request. Working against them are the new models that think at length before answering. These generate hundreds of additional tokens internally that the user never sees. Anyone who offers such models at a flat rate risks their margin.

The metric in quarterly figures and price lists

In the financial reports of large cloud and AI companies, the term rarely appears verbatim. Instead, the companies speak of the gross margin of their AI business. Analysts work backward from this to figure out what a token really costs. Conspicuously low gross margins are seen as a sign that a provider has not yet gotten its compute costs under control.

As a user, you notice the token margin indirectly. It is the reason why free chatbots have message limits and switch to weaker models. It also explains the price wars: between 2023 and 2025, prices per million tokens fell by more than a hundredfold at comparable quality. For developers building their own applications on top of such models, the margin is even directly noticeable. They buy tokens wholesale and must charge their customers more than the responses cost them.