Tail Latency

When you click on something on a website, it takes a short time until a response arrives. This waiting time is called latency. Most of the time it’s very short, but sometimes surprisingly long. Tail latency is exactly this long, rare case: the waiting time of the slowest requests. The name comes from the fact that these cases lie at the extreme edge of a statistical distribution, in the “tail” of the distribution. Operators of large services are often more interested in this edge than in the average.

Why the average lies

Suppose a service responds in 100 milliseconds on average. That sounds good. Nevertheless, every hundredth request might take three seconds. In the mean value, this outlier almost completely disappears. The user, however, experiences it very clearly.

That’s why engineers don’t measure the mean value, but percentiles. The p99 latency is the value that 99 percent of all requests stay within. Only the slowest one percent lies above it. p95 and p99.9 are also common. The further out you measure, the more expensive it usually becomes to improve the value.

The effect intensifies because modern applications break down a single user request into many sub-requests. A product page queries price, stock levels, reviews, and advertising separately. If the page has to wait for all parts, the slowest one always counts. With a hundred sub-requests each carrying a one-percent risk of being an outlier, an outlier is almost guaranteed. A rare problem thus becomes the normal case.

Where the outliers come from

A common reason is shared hardware. Many tasks run simultaneously on a single server. If one of them suddenly needs a lot of computing power, the others have to wait. Background maintenance work also slows things down, such as memory cleanup or writing backup copies.

A second reason is queues. Requests don’t arrive evenly, but in bursts. When a burst hits, requests line up behind each other. Similar to a supermarket checkout: most people get through quickly, but whoever arrives at the wrong moment waits a long time.

There are proven tricks against both. You can send the same request to two servers and take the first response. You can cancel requests that take too long and deliver a substitute value instead. And you can load unimportant parts of a page afterward instead of waiting for them. Measurement itself is crucial: anyone who only logs average values will never see the problem.

Tail latency in chatbots and cloud services

Tail latency is particularly visible in AI services. A language model normally responds after a short time. But if many users are active at once, requests pile up in front of the graphics cards. Then the first letter of the response only appears after several seconds. This exact value shows up in technical reports as “Time to First Token”, measured at the 99th percentile.

Cloud providers also make commitments about this. Contracts for guaranteed service quality often specify percentile values instead of averages. Anyone purchasing an API should therefore check which percentile is meant. A provider with a better average can be clearly worse at p99.

In everyday life, you notice the difference with video streaming, online gaming, and payment processes. Most of the time everything runs smoothly, but occasionally it stutters for a few seconds. These moments shape the impression more strongly than the thousand successful ones before them. A related but different concept is throughput: it describes how many requests a system can handle per second. High throughput and low tail latency are often in conflict, because a fully utilized system builds up longer queues.