Round-Trip Overhead

When your phone sends a request to a remote computer, time passes before any computation even begins. The signal has to travel through cables, cell towers, and distribution stations all the way to the data center. The response has to take the same path back. This pure travel time for the round trip is called round-trip overhead. The word overhead here means: effort that contributes nothing to the result but still occurs. It arises even if the response is ready on the other end within fractions of a second.

Why a response feels slow even though the server is fast

A single round trip within Germany often takes 10 to 30 milliseconds. To a data center in the US, it’s more like 100 to 150 milliseconds. That sounds harmless, and for a single request it is. It becomes a problem once these wait times start piling up.

That’s exactly what happens constantly with modern AI applications. A so-called agent — a program that works through a task independently in multiple steps — doesn’t query the language model just once, but twenty times. In between, it might also call up a search engine and a database. With thirty round trips at 120 milliseconds each, 3.6 seconds have passed on their own, without a single thought having been computed.

A common misconception is that the problem can be solved with more bandwidth. Bandwidth determines how much data fits through per second, not how quickly the first bit arrives. A wider channel helps when downloading large files. Against the waiting time of a short back-and-forth, it practically doesn’t help at all.

What consumes the time along the way

Part of the delay is simply physics. Light in a fiber-optic cable travels at around 200,000 kilometers per second. Frankfurt to California and back is over 18,000 kilometers of cable path, which alone already amounts to nearly 100 milliseconds of base fee. No engineer can optimize this portion away.

The rest arises from intermediate stops. Routers forward the data packet, and each hop costs fractions of a millisecond. Then there’s the connection setup: before data can flow encrypted, client and server first exchange handshake messages. In the past, this required several additional round trips; modern protocols like HTTP/3 need significantly fewer.

That’s why developers rely on two strategies. First, bundling: instead of ten small requests, you send one large one with all the data. Second, proximity: you place servers where the users are and keep connections open instead of establishing them anew for every request. For language models, streaming helps additionally — the response is sent word by word, so the user is already reading while the rest is still being generated.

From video calls to AI agents

In everyday life, you notice the effect during online gaming. The notorious ping is nothing other than a measured round-trip time. At 20 milliseconds, a game feels immediate; at 200 milliseconds, every movement lags behind. In video conferences too, the unpleasant experience of talking over each other arises from exactly this delay.

In tech news, the term today appears mainly in connection with AI tools. When a provider advertises that its assistant runs locally on the laptop, one main argument is the elimination of the round trip. Voice assistants are also under pressure: a conversation only feels natural if less than about 300 milliseconds pass between question and answer. A significant portion of that is consumed by the network path.

To draw a distinction: round-trip overhead is not the same as inference time. Inference time is the model’s own computation duration, while round-trip overhead is the waiting time around it. Anyone wanting to make an application faster should first measure which of the two portions dominates. Often the bigger lever lies not in the model, but in the number of round trips.