Timeout

When a program requests something, it doesn’t know in advance how long the response will take. Maybe it arrives after a tenth of a second. Maybe it never arrives, because the connection is disrupted or the other computer has crashed. So that the program doesn’t wait forever, a maximum waiting time is set in advance. This upper limit is called a timeout. Once it has elapsed, the operation is considered to have failed, and the program does something else.

Why waiting is more costly than giving up

Without a timeout, a single hanging operation can bring an entire system to a standstill. Every request occupies memory and a connection. If a thousand requests get stuck at the same time, eventually all connections are occupied. Then even users whose request would actually have been unproblematic no longer get a response. A small error in one place thus turns into a total outage.

With AI services, there is a second reason: money. A large language model computes on expensive graphics cards. Every second of computing time costs money. A model that gets lost in a very long response burns up capacity that other users are missing out on. Providers therefore set hard limits beyond which generation is cut off.

A common mistake is to simply set the timeout very high in order to avoid errors. This only shifts the problem. A user staring at a blank page for 60 seconds is more dissatisfied than one who sees an honest error message after 3 seconds. Good timeouts tend to be tight rather than generous.

How the clock runs inside the program

Technically, the program starts a counter when the request is sent. If the response arrives beforehand, the counter is cleared. If it runs out, the program triggers an abort. In most programming languages, this is treated as an error that the developer must catch. Typical values range between one and thirty seconds, depending on the task.

Usually a single counter isn’t enough. A distinction is made between how long the connection setup is allowed to take and how long the program then waits for data afterward. A server that doesn’t respond at all should be flagged quickly. A server that is already sending data may be given more time. In addition, there is often an overall limit for the entire request.

After an abort, a second attempt usually follows. So that not all programs knock again at the same time and finally overwhelm the overloaded server for good, each one waits a slightly randomized, growing period of time. This procedure is called exponential backoff. It is also important that the other side likewise stops the aborted task. Otherwise it keeps computing toward a result that nobody will ever pick up.

From the loading indicator to the agent

In everyday life, one encounters timeouts as an error message in the browser: the page loads, and loads, and then a notice appears saying that the connection took too long. There is also a variant of this in online banking. There, one is automatically logged out after a few minutes of inactivity, for security reasons.

With AI products, timeouts appear in two places. The providers' interfaces abort a response if it takes too long. And AI agents, which independently execute multiple steps one after another, are given a budget of time or work steps. Without this brake, an agent could get stuck in a loop and keep working for hours at a cost.

In reports about outages of major services, timeouts are therefore often mentioned. This is rarely the actual cause. It is the visible symptom: somewhere in the system, a component stops responding in time, and the aborts propagate outward until users see an error page.