Checkpointing

When an AI program learns, it modifies billions of internal numerical values over the course of weeks. During this process, these values reside in the fast working memory of specialized computing cards. If the power fails or a machine crashes, this memory is immediately wiped. Checkpointing is the countermeasure: at fixed intervals, the complete intermediate state is written to persistent storage. You can think of it like a save point in a video game. After a crash, you load the last save and only lose the time since the last save.

Why weeks-long training runs would be impossible without save points

Large AI models are trained simultaneously on thousands of computing cards. The more devices involved, the more likely it is that one of them will fail somewhere. In very large setups, this often happens several times a week. Without checkpointing, every single failure would destroy the entire run. A training run spanning two months would then practically never come to an end.

Behind this lie very concrete costs. A large training run can consume several million euros in compute time. If a day is lost, that quickly amounts to a high five-figure sum. Checkpointing limits this potential loss to the interval between two save points. If you save hourly, you risk at most one hour of work.

Checkpoints have a second benefit that is often underestimated. You can later compare how good the model was after 30, 60, or 90 percent of training. Sometimes an earlier state is even better than the final one. In that case, you simply take that checkpoint as the finished model. Without saved intermediate states, this kind of fallback would not be possible.

What is actually contained in a save point

A checkpoint contains more than just the model’s learned numerical values, the so-called parameters. It also stores the state of the learning procedure that adjusts these values. In addition, there is information about exactly where one currently stands in the training data. Only with all three components does training continue seamlessly. If one of them is missing, the model shows a visible drop in quality after the restart.

Such files are enormous. A model with 70 billion parameters already takes up about 140 gigabytes as a pure parameter file. Including the state of the learning procedure, this easily becomes three to four times as much. Writing this amount of data takes minutes, during which training is paused. That is why one does not save constantly, but instead seeks a compromise.

This is exactly where the real trade-off lies. If you save too rarely, you lose a lot of work in the event of a crash. If you save too often, the system spends a noticeable portion of its time writing instead of computing. Modern systems solve this by writing in the background while training continues. Frequently, only the last few checkpoints are kept, with older ones deleted automatically.

From the training data center to the download portal

Anyone who downloads a freely available AI model is actually downloading a checkpoint. On platforms like Hugging Face, the files are often named exactly that. They contain the frozen learning state of a model at a specific point in time. Labels with additions like “final” or a step count indicate from which phase of training the state originates.

The term appears in corporate announcements when discussing data center outages. Reports on large training runs regularly mention how often hardware failed and how much time was lost due to restarts. A common misconception, incidentally, is that checkpointing is the same as a backup. A backup protects data against loss, whereas a checkpoint protects a running process against interruption. The purpose is similar, but the content is entirely different.