Latency

Latency is the time that passes between a request and the first reaction to it. You press a button, and it takes a moment before something happens. That exact moment is the latency. It is measured in milliseconds, meaning thousandths of a second. Latency is not the same as the amount of data flowing through per second: a connection can transport a great deal and still take a long time before the first byte arrives. A comparison helps: a freight train carries a lot of cargo but is slow to get going, a motorcycle courier carries little but arrives instantly.

Why milliseconds decide money and feeling

People notice delays earlier than one might think. Below about 100 milliseconds, a reaction feels instant. From half a second on, one senses a hesitation. From two to three seconds, many users give up and try something else. That's why latency is not a technical nicety for online services, but a business risk.

In stock trading this becomes extreme. Computer programs there buy and sell automatically, and whoever sees a price change milliseconds earlier can react sooner. Trading firms therefore rent server space directly next to the exchange's computers so that the cables are shorter. Millions are spent for the sake of a few milliseconds.

With AI services a second point comes into play. A language model, meaning a program that generates text word by word, has to compute anew for each word. If the initial delay is already large, the whole conversation feels sluggish. Companies therefore often report two values: the time to the first word and the time between the following words.

What the waiting time is made up of

Latency is almost always a sum of several partial stretches. First, the signal needs time to travel through cables and radio links. Light in a fiber-optic cable is fast, but not infinitely fast: for the route Frankfurt–New York and back, about 80 milliseconds are spent on the travel alone. Nobody can program this physical limit away.

Then comes the waiting time in the devices along the way. Routers and switching stations accept data packets, check them, and forward them. When many packets arrive at once, a queue forms, similar to a checkout line at a supermarket. Such congestion is the main reason why latency fluctuates instead of staying constant.

At the destination itself, computing time arises. A server has to understand the request, perhaps query a database, and produce a result. With AI models, this part is often the largest, because billions of computing operations occur in the process. Remedies include faster specialized chips, smaller models, and caches in which frequent answers are already stored ready-made. Another strategy is called edge computing: you place computers closer to the users so that the path becomes shorter.

Latency in gaming, video calls, and product promises

Latency is most clearly noticeable in online gaming. The ping displayed there is exactly this measurement: the time for the path to the game server and back. At 20 milliseconds, the game character reacts as if instantly; at 200 milliseconds, you shoot into thin air. Similarly with video calling: as soon as the delay grows large, both sides talk over each other.

In product announcements, latency often appears as a selling point. Mobile network providers advertise low latency in 5G networks, chip manufacturers advertise faster response times of their processors. For voice assistants, low latency is the prerequisite for a conversation feeling natural, because in real dialogues people pause only fractions of a second.

A common misconception: a faster internet connection automatically lowers latency. More bandwidth helps when downloading large files, but hardly changes the waiting time until the first reaction. Whoever has poor latency usually has a problem with distance, with overloaded lines, or with a slow server on the other end.