Pre-Training Run

A program that can write texts or answer questions is not programmed by humans rule by rule. It learns on its own by reading an unimaginable amount of text while constantly predicting which word comes next. This first major learning phase is called pre-training. A single run of this process is a pre-training run. It is started once, then runs for weeks or months, and ends with a finished base model. Everything that follows only builds on this result.

The most expensive single line item of an AI project

A large pre-training run is perhaps the riskiest expenditure in the tech industry. For this, companies rent or buy thousands of graphics chips, so-called GPUs, which are especially well suited to such computing tasks. The costs for a single run of the largest models range in the tens or hundreds of millions of dollars. On top of that comes electricity consumption on the scale of a small city.

The tricky part: you only see at the end whether the result is good. An error in the data or in the settings can destroy weeks of computing time. That’s why teams first test their ideas on small models and extrapolate what should happen with the large version. These extrapolations are called scaling laws.

This is also relevant for the stock market. When a corporation announces it will invest several billion dollars in new data centers, a large portion of that goes toward upcoming pre-training runs. The chipmaker Nvidia earns its money primarily from exactly this demand.

From a mountain of text to a base model

At the start is the dataset: websites, books, program code, forums, often several trillion words. This mountain is filtered, cleared of junk and duplicates, and broken down into small units called tokens. A token is usually part of a word, roughly a syllable. The model starts with pure randomness in its millions or billions of adjustable values, the parameters.

Then the actual loop begins. The model reads a text excerpt, predicts the next token, and compares its prediction with the real word. The gap between prediction and truth is the error. Afterward, all parameters are shifted a tiny bit in the direction that makes this error smaller. This loop repeats many millions of times.

A comparison helps: it’s like a student who reads an entire library shelf, but never memorizes facts, only develops a feel for language and connections. That’s exactly why a base model cannot yet give polite, helpful answers right after the run. It merely continues text in a plausible way. The response behavior only comes from a later, much shorter post-processing step, fine-tuning.

How to recognize pre-training in the news

When a new model generation is introduced, the term almost always comes up. Sentences like “trained on 15 trillion tokens” or “knowledge cutoff mid-2024” refer to the pre-training run. The knowledge cutoff is simply the point in time at which the collection of training data ended. Everything that happened in the world afterward is something the model does not know on its own.

Legal disputes also hinge on this. Lawsuits by publishers and artists against AI companies mostly revolve around texts and images that were fed into such runs without permission. Since a run cannot be undone, individual works can hardly be removed from the model afterward.

A common misconception is that a chatbot keeps learning during a conversation. It does not. The pre-training run is long since finished, the model is frozen. New information comes either from the ongoing conversation, from a web search, or from a completely new run for the next model version.