Checkpoint

An AI model learns by working through millions of examples over the course of weeks. Its entire knowledge is stored in a huge collection of numbers, known as parameters. These numbers change a little bit constantly during learning. A checkpoint is a copy of all these numbers, captured at a specific point in time and written to disk. You can think of it like a save point in a video game: you save your current progress so you can pick up exactly where you left off later. Without such save points, any mishap during training would mean a total loss.

Why weeks-long training would be impossible without save points

Large models are trained on thousands of graphics cards simultaneously. With this amount of hardware, something is practically always failing. A card overheats, a network cable comes loose, the power supply fluctuates. Without intermediate states, you would have to start from zero after every such incident. For a run lasting two months, that would simply not be feasible.

Compute time is also very expensive. A large training run can consume several million euros in electricity and data center rental costs. A lost day therefore costs real money. Checkpoints limit the damage to the time since the last save, usually just a few hours.

There is also a second reason. A model does not always just keep getting better while learning. Sometimes the learning process derails, and results suddenly get worse. In that case, developers fall back on an older checkpoint and restart from there with changed settings. The save point is therefore not just a safeguard against failures, but also a tool for experimentation.

What such a file actually contains

A checkpoint first contains the parameters, i.e. all the learned numerical values of the model. For a large language model, this quickly adds up to hundreds of billions of values. Every value requires storage space, which is why such files can be several hundred gigabytes in size. For comparison: that is more than fits on many laptop hard drives.

However, this is not enough for training to be resumable. Also saved is the state of the optimizer, i.e. the procedure that decides how strongly each number is adjusted at each step. In addition, there is the number of the current training step and the position within the dataset. Only with all this information does training continue seamlessly, as if nothing had happened.

The writing process itself takes time and blocks the computation. That's why saving doesn't happen constantly, but at fixed intervals, for example every few hours. A distinction is often made between full checkpoints for continuing computation and leaner versions that contain only the parameters. The latter are sufficient if you only want to use the model rather than continue training it.

Checkpoints in downloads, releases, and technical reports

Anyone who downloads a freely available model from the internet is, in truth, downloading a checkpoint. On platforms such as Hugging Face, several states are often available for one model. They differ in how long training lasted or which additional task the model has been further tailored to. With image generators, users regularly exchange different checkpoints to obtain different image styles.

The term also appears in news reports. When a company writes that it is releasing an intermediate state of its model, that is exactly what is meant: a snapshot, not the final product. Some research groups deliberately release many intermediate states so that others can trace when the model acquired which capability.

A common misconception is that a checkpoint is the same as the finished model. More precisely, a model is the architecture, i.e. the arrangement of the computational building blocks. The checkpoint provides the numbers that actually make this architecture usable in the first place. Without a matching architecture, the file is just a meaningless collection of numbers. Only both together produce a working AI.