Client Timeout

When a program fetches data over the internet, it waits for a response. The requesting program is called the client, for example an app on a phone or a browser. The responding computer on the other end is the server. A client timeout is the deadline the client sets for itself: if no response arrives by then, it stops waiting and reports an error. Typical values are 30 or 60 seconds, but sometimes as little as 5. Without such deadlines, a program could hang forever, waiting for a response that never comes.

Why aborted requests still cost money

A timeout primarily protects the user. An app that says “Connection failed” after half a minute is better than one that shows a loading bar forever. The user can try again or do something else. Without a timeout, resources in the client are also tied up: open connections, occupied memory, waiting processes.

With AI services, however, the cancellation has an unpleasant side effect. The server often doesn’t notice right away that no one is listening anymore. It finishes computing the answer even though it goes nowhere. The computing time on expensive graphics cards is thus used up, but the result is worthless. With many users at once, this adds up to a noticeable cost item.

A timeout is even more delicate for requests that change something. If a payment or an order is aborted, the client doesn’t know whether it went through. The server may have already executed it. A retry can then lead to a duplicate booking. Well-built systems prevent this by having every request carry a unique identifier, so that sending it multiple times still only takes effect once.

Who sets the clock and where it ticks

The clock runs in the client, not in the network and not in the server. As soon as the request goes out, a counter starts. Once it reaches the set limit, the program throws an error and closes the connection. Think of it like a kitchen egg timer: it rings regardless of whether the cake is actually done.

In practice, there are usually several such clocks in sequence. One for establishing the connection, one for waiting for the first line of the response, one for the entire transfer. For chat systems, the last one is especially important. Such systems typically respond via streaming, meaning word by word instead of all at once at the end. A tight overall timeout then cuts off long responses in the middle of a sentence.

It’s important to distinguish this from a server timeout. There, the server itself sets a limit and terminates a request that’s taking too long from its side. Anyone who sets both deadlines without coordinating them ends up with hard-to-explain errors. As a rule of thumb, the client’s deadline should be somewhat longer than the server’s, so that the server can cleanly report that it’s giving up.

From the loading bar to the error message in the API

In everyday use, a client timeout shows up as a short, matter-of-fact error message. “Request timed out,” “The operation was cancelled,” or a loading spinner that eventually disappears. This happens frequently on poor mobile networks, because responses take longer there than the client allows.

Anyone writing their own code sets these limits explicitly. The programming interfaces of major AI providers — that is, the technical access points through which other programs use the model — document their own recommendations for this. For elaborate requests to so-called reasoning models, which go through longer computation steps before responding, several minutes are recommended. Anyone who leaves the default value of the program library as is will see cancellations across the board.

A common misconception is: a timeout means the server has crashed. That’s rarely true. Usually it was just overloaded, or the response simply took a long time. That’s why many systems automatically retry the request, but with increasing pauses. This procedure is called backoff, and it prevents thousands of clients from hammering away simultaneously and finally bringing the already slow server to a complete halt.