Response Time

Response time is the span between an input and the system’s response. You press Enter, and at some point the first letter appears on the screen. That exact waiting period is what’s meant. It’s usually measured in milliseconds, i.e. thousandths of a second. For a voice assistant, it’s under half a second in the best case. For a large text model answering a complicated question, it can take several seconds. Important: response time says nothing about whether the answer is also correct. It only measures speed.

Why half a second decides success

Humans notice delays with remarkable precision. From about 100 milliseconds onward, a reaction no longer feels instantaneous. From about one second onward, people start to get bored or distracted. In spoken dialogue, the threshold is even stricter, because we’re used to very short pauses from real conversations. An AI assistant that stays silent for two seconds therefore quickly seems sluggish.

For companies, this isn’t a cosmetic issue but a business risk. Large online retailers have measured that even a 100-millisecond delay per page load measurably reduces revenue. The same applies to AI products: users abandon a session if the answer takes too long. In some areas, delay is even dangerous. A self-driving car has to detect an obstacle and brake before it’s too late.

Response time shouldn’t be confused with throughput. Throughput means how many requests a system can handle per second overall. A data center can have high throughput and still be slow for each individual user. Optimizing both are two different tasks that sometimes even conflict with each other.

What the waiting time consists of

Response time is always the sum of several parts. First, your request travels through the network to a remote data center. This journey alone takes time, since even light signals in fiber-optic cables need a few milliseconds to cover a thousand kilometers. Then the request may wait in a queue because many other users are active at the same time. Only after that does the actual computation begin.

For language models, two values are distinguished. The first is the time until the first character of the answer appears. The second is the pace at which the remaining words follow. That’s why chatbots write their answer onto the screen piece by piece instead of waiting until everything is finished. This considerably shortens the perceived waiting time, even though the total duration stays the same.

Response time can be shortened in several ways. Servers are placed closer to users so the network path becomes shorter. Smaller models are used for simple requests, and large ones only for difficult ones. And frequent answers are cached instead of being recalculated every time. A common misconception is that a stronger processor always helps. If the delay is mainly in the network path, more computing power does almost nothing.

Where you notice this topic in news and products

In marketing texts from AI companies, the English word latency is often used when response time is meant. Voice modes of assistants are advertised with figures like 300 milliseconds, because that roughly corresponds to a natural conversational pause. With the 5G mobile standard too, low response time was a central selling point, not just the high data rate.

In everyday life, you notice the difference especially when playing online games. There, response time is usually called ping, and a high ping means your shot arrives late. In video chat, high response time causes conversation partners to constantly interrupt each other. And when a chatbot thinks for minutes, there’s often a so-called reasoning model behind it, which deliberately takes longer to compute in order to give better answers. Here, the long response time isn’t a flaw but a deliberate trade-off.