Retry Logic

Programs are constantly asking other computers for data. A weather app asks a remote server for the temperature, a banking app asks for the account balance. Sometimes no answer comes back: the connection stalls, or the other server is momentarily overloaded. Retry logic is the built-in rule to automatically try again in such a case, without any human intervention. It answers three questions: How many times will it retry? How long will it wait between attempts? And when does the program finally give up and report an error?

Why a second attempt is often enough

Many network disruptions are temporary. A server is briefly overloaded for half a second, a data packet gets lost along the way, a connection is re-established. Such errors are called transient, meaning fleeting. If you wait just one second and try again, you’ll often get a clean response.

Without retry logic, every one of these mini-disruptions would reach the user directly. The app shows an error message even though nothing is actually broken. For large services, this adds up: if a single request only succeeds 99.9 percent of the time, out of a million requests per day, a thousand still fail. An automatic second attempt pushes that number down to a fraction.

This is especially important for AI services. A language model responds via an interface on the internet, meaning an address to which a program sends its request. These interfaces are often heavily loaded and reject requests with a “too many requests” message. Programs that use such AI services therefore almost always need retry logic, otherwise they would constantly fail during everyday operation.

Exponential backoff and the random jitter

The naive approach would be to ask again immediately and repeatedly. But this makes the situation worse. If a server is overloaded, even more requests now come pouring in on it. That’s why exponential backoff is used: the wait time doubles after each failed attempt. So first wait one second, then two, then four, then eight. After about five attempts, the program stops and honestly reports the error onward.

On top of that comes a random addition, known in technical jargon as jitter. Imagine a thousand phones experiencing a disruption in the same second. Without randomness, they would all retry again at exactly the same moment one second later, knocking the server over a second time. With a random addition of a few hundred milliseconds, the attempts get spread out cleanly over time.

A common misconception is that you can retry everything. That’s not true. Retrying only makes sense for transient errors. But if the response is “incorrect password” or “this address doesn’t exist,” even the tenth attempt won’t change anything. It gets dangerous with actions that change something, such as a bank transfer. If the first request actually arrived and only the confirmation was lost, the second attempt will book the payment twice. A transmitted identifier helps here, letting the server recognize that it’s the same order.

From the loading bar to the outage report

In everyday life, you usually only notice retry logic by not noticing it. The loading bar spins a second longer than usual, then the video appears. In the background, the third attempt may already have run. Streaming services, messengers, and online games also constantly work with such retries.

In tech news, the term usually comes up after major outages. In the post-incident analyses, you then read that poorly configured retry logic prolonged the outage. The mechanism is called a retry storm: a service wobbles, millions of clients immediately try again, and the flood of retries keeps the service down even though the original problem has long since been fixed.

Anyone who codes themselves rarely needs to write retry logic by hand these days. Ready-made program libraries and the official access packages from major providers already include it. You just need to set how many attempts are allowed and how long to wait. Related, but not the same, is the circuit breaker: it stops requests to a service recognized as dead entirely, instead of continuing to retry.