Failure Recovery

Computers fail. A hard drive gives up the ghost, a program crashes, an excavator severs a cable. Failure Recovery is the technical term for everything that happens afterward to get the system working normally again. This includes retrieving data thought to be lost and either cleanly completing or rolling back work that was in progress. The term does not refer to repairing broken hardware, but to the planned return to a reliable state. What matters is that this return is designed in advance, not improvised in an emergency.

Why failures are planned for instead of avoided

A single server rarely fails, perhaps once every few years. But large data centers operate hundreds of thousands of machines simultaneously. At this scale, something breaks statistically every single day. Providers like Google or Amazon therefore do not assume everything will work. They build their systems from the outset so that a failure goes unnoticed by anyone.

The economic damage makes the difference tangible. If a trading platform is down for an hour, it loses revenue in the millions. At a bank, even a single incorrectly aborted transfer can become expensive. That’s why companies negotiate two fixed metrics in contracts. One states the maximum time a system may be down; the other states how many minutes' worth of data may be lost at worst.

A common misconception is that a backup copy alone is enough. A copy that has never been restored is just an assumption. Many companies only discover in an actual emergency that their backup was incomplete or unreadable. That’s why regular recovery drills are an essential part of the topic.

Logs, copies, and the point you jump back to

The most important tool is a record of all changes, known in technical jargon as a log. Before a database actually carries out a change, it writes down what it intends to do. If it crashes partway through, it reads this log upon restart. It rolls back half-finished operations and replays those that were already confirmed. This produces a state in which no transfer was ever left only half-completed.

In addition, systems create intermediate states at regular intervals, known as checkpoints. Think of them like a save point in a video game. Without a save point, you would have to replay the entire log from the very first day. With a save point, only the last short stretch is needed, and recovery takes seconds instead of hours.

The third ingredient is redundancy, meaning deliberate duplication. Important data is stored on multiple machines in different buildings. If one fails, another takes over immediately. This is precisely where Failure Recovery diverges from a related concept: fault tolerance means the failure is never even noticeable in the first place. Recovery describes the path back once it has become noticeable.

From cloud outages to interrupted AI training

The topic usually appears in the news after major outages. When Amazon Web Services or Microsoft Azure go down for several hours, thousands of apps are simultaneously offline. The providers' subsequent reports explain exactly which mechanism failed and how long recovery took. Such reports are public and an important part of trust in cloud services.

Recovery also plays a central role in training large AI models. Such a training run occupies thousands of graphics cards over weeks. If just a single one fails, the entire run effectively grinds to a halt. That’s why the systems save an intermediate state of the model every few minutes. After a disruption, training resumes from that point instead of starting from zero.

You encounter the same principle in everyday life, on a smaller scale. A word processor that offers the last version after a crash is working with intermediate states. A phone that automatically loads the old version after a failed update is performing a rollback. The only difference from data centers is the scale, not the underlying idea.