Serving costs

When a company offers an AI chat or a translation feature, that feature has to be available around the clock. To make this happen, special computers run in data centers that process every incoming question and send back an answer. This provisioning and answering is called serving in the industry. The costs for this are called serving costs: electricity, rent for the machines, network, staff for operations. Unlike the one-time effort of getting a model to learn from example data in the first place, serving costs are incurred continuously. They keep growing with every user.

The bill that keeps running with every answer

With classic software, the cost of an additional user is close to zero. A word processor can be copied millions of times without each copy costing computing time. With AI services, it’s different. Every answer has to be freshly computed, and every computation costs money.

That’s why serving costs are one of the most important figures for investors when it comes to AI companies. A provider can have millions of subscribers and still make losses if every request costs more than it brings in. Flat-rate plans are especially tricky: anyone who can ask unlimited questions for a fixed monthly price can turn into a losing proposition for the provider. This is exactly why terms like gross margin show up in quarterly reports in connection with AI products.

There’s one potential mix-up worth knowing about. Training costs are a large, visible investment that gets a lot of coverage. Serving costs are less conspicuous, but over the years they often add up to a larger amount.

What drives down the price per request

The biggest cost factor is graphics chips, so-called GPUs. These are specialized processors that can carry out a great many calculations simultaneously. They are expensive to buy and consume a lot of electricity. Anyone who leaves such chips sitting around unused is burning money. A central goal in operations is therefore to keep the chips as fully utilized as possible.

A common trick is batching requests. Instead of computing one question at a time, the system briefly collects several questions and processes them together. This makes better use of the chip and lowers the cost per answer. The price for this is a small delay for the individual user. Operators therefore have to weigh cheap against fast.

Other levers target the model itself. Numbers within the model can be stored more coarsely, which saves computing time and only slightly changes answer quality. Smaller models can be used for simple questions, passing only difficult cases on to the large, expensive model. And frequent answers can be cached so that the same calculation doesn’t run twice. In recent years, the cost per request has fallen sharply thanks to methods like these.

Serving costs in price lists and quarterly figures

They are most directly visible in the price lists of AI providers. There you’ll find a price per million tokens, that is, per million text fragments that sentences are broken down into. These prices are nothing other than serving costs passed on with a markup. Anyone comparing the lists over two years can clearly see the cost curve pointing downward.

In the news, the term usually comes up in two contexts. First, when corporations announce billions in spending on data centers. Second, when analysts ask whether an AI offering is economically viable. Reports about usage limits, slower free versions, or discontinued features often have their root cause here as well.

As a user, you notice serving costs indirectly. Free versions get smaller models or a request limit. Paid versions deliver the more expensive models. And when a service responds more slowly during peak times, a fully utilized data center is often behind it.