Latency Guarantee

When a computer service receives a request, time passes before the response comes back. This waiting time is called latency. For a request to an AI system, it can take a few milliseconds or several seconds. A latency guarantee is a fixed promise from the provider stating the maximum length this waiting time may take. It is usually stated in a contract between the provider and the paying customer. If the commitment is broken, the provider must answer for it, for instance with a credit.

When milliseconds decide over money

For many applications, a slow response is as bad as no response at all. A trading system on the stock exchange must react within fractions of a second. A voice assistant in a car seems broken if it stays silent for three seconds. A fraud filter must decide before the card payment goes through. In all these cases, speed is not a convenience but the actual function.

That is why it is not enough for companies that a service is usually fast. They need a commitment they can plan around. Without such a commitment, they would have to buy and operate their own servers. The latency guarantee is thus the argument with which cloud providers win larger customers in the first place. Here, cloud means: computing power that one rents from an external provider instead of owning it oneself.

It is important to distinguish this from availability. Availability states how often a service can be reached at all. Latency states how quickly it responds when it is reachable. A service can run 99.99 percent of the time and still be agonizingly slow. That is why both values are guaranteed separately.

Why guarantees almost always work with percentages

No provider promises a maximum for every single request. A single network error would otherwise break the commitment immediately. Instead, so-called percentiles are used. A typical wording reads: 95 percent of all requests are answered in under 300 milliseconds. The remaining five percent may be slower without the contract being violated.

Providers keep such commitments not through good intentions but through reserves. They hold more computing capacity ready than would normally be necessary. This reserve absorbs load spikes, for example when all users are active at the same time in the evening. In addition, servers are located geographically close to users, because the signal on the line also takes time. If it still gets tight, less important requests are pushed into a queue.

With AI models, there is an additional peculiarity. A language model generates its answer word by word, so the total duration depends on the length of the response. That is why the time until the first word is often measured separately from the speed afterward. A guarantee covering the entire response time would otherwise be unfair or meaningless.

Latency guarantees in everyday life and in the business news

As a private user, one rarely signs such contracts oneself. Nevertheless, one notices their effect. A video stream that starts without stuttering, or a navigation app that instantly recalculates the route, depend on such commitments in the background. Online games also advertise guaranteed response times of their servers.

In the news, latency guarantees usually appear under the abbreviation SLA. SLA stands for Service Level Agreement, i.e. an agreement about the guaranteed quality of service. When a cloud provider announces new AI data centers, it is almost always also about tighter commitments of this kind. Providers that can guarantee faster response times charge significantly higher prices for it.

A common misconception is that a latency guarantee is a technical property of the system. It is first and foremost a legal promise with a penalty in the fine print. This penalty is also usually limited, often to a portion of the monthly bill. Anyone who loses a lot of money because a service was too slow rarely gets the full damage reimbursed.