Response Time

You type a question into a chat window and press Enter. Then a moment passes before the first line of the answer appears. This exact span is called response time. It’s usually measured in milliseconds, i.e. thousandths of a second. For programs that deliver an answer all at once, the matter is simple: you measure from sending to the complete result. For language programs that output their answer word by word, two things are distinguished — how long the first word takes, and how quickly the following ones follow.

Why seconds decide usage

Humans are remarkably sensitive to waiting times. Below about 100 milliseconds, a reaction feels instant. Up to around one second, the train of thought remains undisturbed. From about ten seconds onward, many users turn away or switch tabs. For providers, response time is therefore not a technical footnote but a business metric.

With AI services, a second point comes into play. Short response times are bought with computing power, and computing power costs money. A provider can deploy more specialized chips so that no one gets stuck in a queue. But these chips then sit unused during quiet hours. So there is a real trade-off between fast responses and low costs.

It’s also important that an average value says little. Experts therefore look at so-called percentiles. The value P95 means: 95 percent of all requests were faster than this value. If the average is one second, but the P95 is twelve seconds, then many users regularly experience agonizingly long wait times.

What the waiting time is made up of

Response time is a sum of several partial legs. First, your request travels across the network to the data center, which takes anywhere from a few dozen milliseconds depending on distance. There it often lands in a queue because other requests are currently being processed. Only then does the actual computing work begin on the chips. Finally, the finished answer must travel the same path back.

For a language model, the computing work itself splits into two phases. First, the model reads in your entire input and processes it. This explains why a question with ten pages of attached text noticeably starts more sluggishly than a short question. After that, the model generates the answer piece by piece, with each new word requiring its own computation pass.

Developers have several levers. Frequently asked questions can be cached, so the answer is ready without recomputation. Smaller models respond faster but are less capable. And servers closer to users shorten the path through the network. A common trick is also outputting the answer step by step: the total duration doesn’t change as a result, but the perceived waiting time drops significantly.

Response times in products and headlines

In everyday life, you encounter response time everywhere software reacts to an input. A search engine delivers results in a fraction of a second. A voice assistant in a car shouldn’t stay silent for long after a command. In online gaming, people talk about ping — that too is nothing other than a response time, just between your device and the game server.

In reports about AI providers, the term usually appears in comparisons. Companies advertise that their model responds faster than the competition’s. Independent test sites continuously measure such values and publish rankings. There, response time almost always appears alongside the price per request and the quality of the results.

A common mistake is confusing response time with throughput. Throughput describes how many requests a system can handle in total per second. Increasing both at the same time is difficult: bundling many requests together and computing them jointly increases throughput — but the individual request then waits longer as a result. Large providers are constantly searching for the right middle ground here.