Retry Storm

When a program on the internet requests something and gets no response, it usually just tries the request again. That’s normally sensible, since many disruptions last only a fraction of a second. It becomes problematic when thousands of programs fail at the same time and all retry simultaneously. Then a multiple of the normal load hits a service all at once, one that is already struggling. This exact avalanche of retry attempts is called a retry storm. By then, the original error is often long since fixed, but the service still can’t come back up.

When a one-second glitch becomes a day-long outage

A retry storm turns a small error into a big one. Without retries, a brief overload would pass within a few seconds. With them, a vicious cycle emerges: the service is slow, so everyone retries, which makes it even slower. Experts call this a self-reinforcing effect.

What’s especially unpleasant is that the system doesn’t recover on its own. Even if you fix the actual root cause, millions of retries keep pouring in. Operators then have to actively lock out users just to catch their breath. This is called a blocked restart, and it often costs hours.

This is economically relevant because modern software consists of many small services that call each other. If one component fails further down the chain, all the services above it retry. With five layers and three attempts each, a single request can multiply into as many as 243. Major outages at cloud providers have repeatedly been explained by exactly this mechanism.

Why waiting and randomness help

The remedy is called exponential backoff. After the first failed attempt, the program waits about one second, after the second two seconds, then four, then eight. So the intervals double instead of staying constant. This causes the load to decrease with each round instead of remaining the same.

Added to this is a random component, known in technical jargon as jitter. Without it, all programs would wait exactly the same amount of time and hit the service again in lockstep. You know this effect from a traffic light: when it turns green, everyone starts driving at once. A random delay of a few hundred milliseconds spreads the requests out over time.

A third building block is the circuit breaker, essentially a kind of fuse like in a breaker box. If errors pile up, it immediately blocks further requests and returns an error right away. This gives the overloaded service some peace to recover. After a while, the fuse lets a few requests through as a test, and only then switches back on fully. Also important is a hard cap: three attempts are almost always enough, and endless retrying never helps.

From ticket sales to outage reports

Users experience retry storms without even noticing them. When tickets for a sought-after concert go on sale, tens of thousands hit refresh at the same time. Each of these clicks is a manual retry attempt and makes the situation worse. The same thing happens with university enrollment portals or with tax software just before the deadline.

In the news, the term usually shows up in the reports cloud providers publish after an outage. Such post-mortems, meaning after-the-fact analyses, almost always cite a harmless root cause, followed by the retry avalanche as the actual amplifier. This matters to investors because downtime at providers like Amazon, Microsoft, or Google directly triggers contractual penalties.

A common misconception is that retry storms can be solved by adding more servers. Additional capacity is often devoured by the avalanche within seconds. The only effective fix is to limit the number of retries itself. The term should be distinguished from the thundering herd: there, everyone rushes at once toward a resource that has just become available, without anything having failed beforehand.