Training Run

A program for artificial intelligence is not written line by line by humans. Instead, it is presented with vast amounts of examples: texts, images, program code. From these examples, it derives regularities on its own, adjusting millions of internal parameters in the process. This process is called training, and a single continuous pass of it is a training run. It has a clear beginning, a clear end, and a result: a finished file containing the set values. This file is what is later used as the model, for example as a chat program.

Why a single run decides company fates

A large training run is one of the most expensive single actions in the tech industry. For the flagship models of well-known providers, amounts between one hundred million and over a billion dollars are cited. The money flows into compute time on specialized chips and into electricity. It is spent before anyone knows whether the result will be good.

This is exactly where the risk lies. If a run produces a weak model after two months, the money is gone. You cannot simply correct the last three days. Often you have to restart from an earlier intermediate state or begin all over again. That is why research teams run many small, cheap test runs beforehand.

For investors, the term matters because it explains where the AI industry’s capital is heading. The enormous investments in data centers and graphics cards largely serve to make such runs possible in the first place. Anyone talking about the valuation of chip manufacturers is indirectly talking about planned training runs.

What happens during the weeks in the data center

At its core, the process is a loop that is repeated very many times. The model is given a slice of the data and makes a prediction, for instance which word comes next. The system then compares the prediction with the actual next word. From the deviation, it calculates in which direction the parameters need to be shifted. Afterward, the step begins again with the next slice of data.

Because a single chip would need decades for this, the work is distributed across thousands of chips. They must coordinate after every step, otherwise their parameters would drift apart. As a result, the network between the chips is often the bottleneck, not the computing power itself. If one chip fails, the entire run can stall.

For this reason, the system regularly saves intermediate states, so-called checkpoints. After a crash, work resumes from there instead of everything being lost. The teams also continuously monitor a metric for the average error. If it stops decreasing or suddenly jumps upward, humans step in.

The term in news reports and product names

In the news, the training run usually appears as a time reference. Sentences like “training of the next model has been running since fall” are a hint at when a product might be released. Reports about aborted or delayed runs are also common. They are seen as a sign that a provider is having technical problems.

You can also see the traces of a run directly in the product. When a chat program says its knowledge cutoff is a certain date, that is due to the data of the training run. The model does not know about events after that, because they were not part of the examples. Newer systems get around this by additionally looking things up on the internet.

A common misconception is that models keep learning during a conversation. They do not. A chat does not change the parameters; a new training run would be required for that. What is done afterward are smaller follow-up trainings, for example to make responses more polite or safer. These are considerably cheaper than the large initial run.