Serving Report

Once an AI system is fully built, the real operation begins. Users send requests, the system computes, and sends answers back. This ongoing operation is called “serving” in technical jargon, meaning the delivery of answers. A serving report is the written evaluation of this operation over a specific period, usually a day, a week, or a month. It states how many requests came in, how long the responses took, what they cost, and how often something went wrong. You can think of it like the monthly report of a power plant: not how the facility was built, but how reliably it actually runs.

What operation really costs

Training an AI model is a one-time, expensive investment. Operating it for years afterward is an ongoing burden. Every single response consumes computing time on expensive specialized chips. With millions of requests per day, cent amounts add up to six-figure bills. The serving report is often the only document that makes this sum visible at all.

That’s why not only engineers are interested in it, but also controlling and management. A company can use the report to decide whether an AI feature pays off. If a feature costs 40 cents per user per month, but the subscription only brings in 5 euros, that is hard business information. Conversely, the report also shows where optimizations have actually had an effect.

A second reason is reliability. Many providers contractually promise their customers a certain availability, such as 99.9 percent. The serving report provides proof of whether this promise was kept. Violations can lead to refunds or contractual penalties.

Which metrics it contains

The most important figure is usually latency, i.e., the time span between question and answer. It is rarely given as an average, since that obscures outliers. Instead, the so-called P95 value is common: the time that 95 out of 100 requests fall under. If this value is three seconds, five percent of users wait even longer. It is exactly this minority that complains.

Alongside this is throughput, meaning how many requests per second were handled. Then there is the error rate: the proportion of requests that were aborted or received an error message. For language models, the number of processed text chunks, so-called tokens, is additionally counted. From this number, costs follow almost directly, since computing power is usually billed per token.

This data is collected automatically. Every request leaves an entry in a log, and an analysis tool aggregates these entries. The report itself is thus usually not a handwritten text, but an automatically generated dashboard with charts. It should not be confused with an evaluation report: that checks how good the answers are in terms of content. The serving report only checks whether and how quickly an answer was given at all.

Who reads such reports

In everyday life, one rarely encounters serving reports directly, as they remain internal to companies. However, their consequences are felt immediately. If a chatbot responds more slowly in the evening than in the morning, that shows up in such a report the next day. And if a provider suddenly introduces a usage limit, a cost item in the report was usually the trigger.

Parts of it become publicly visible on the status pages of major providers. There, companies like OpenAI, Google, or Amazon publish whether their services are currently running normally. This is a heavily simplified, public version of the same data. The aggregated figures also appear in the quarterly results of tech companies, usually as justification for rising data center expenditures.