Pretraining

A language model like ChatGPT doesn’t come into being in a single step. At the very beginning stands pretraining. During this phase, the program reads enormous amounts of text from the internet, from books, and from programming code. The task is always the same: a piece of text is shown, the next word is hidden, and the program has to guess it. It compares its guess with the actual word and adjusts its internal computational values a little so that it guesses better next time. This process is repeated trillions of times. In the end, it has learned grammar, facts, translations, and writing styles along the way, without anyone ever explaining a single rule to it.

Why pretraining is the expensive part

Pretraining consumes the largest share of money and computing time. Large models run for weeks or months on tens of thousands of specialized chips to accomplish this. The cost of a cutting-edge model runs into the hundreds of millions. That’s why only a few companies and states can even build such a model from scratch at all.

This is exactly where the economic appeal comes from. A pretrained model can be reused again and again. Whoever owns it can turn it into a chatbot, a coding assistant, or a translator. This adaptation costs only a fraction of the pretraining. Experts therefore speak of a foundation model, on top of which many products are built.

The limits of a model are also set at this stage. Whatever is missing from the training text, the model will later barely know. And whatever happens in the world afterward, it has no idea about, because the data has a cutoff date. This so-called knowledge cutoff is the reason why models often get things wrong about current events.

Prediction as a learning principle

Technically, the model consists of billions of numbers, the parameters. You can imagine them as adjustment screws. During pretraining, after each guessing error, a portion of these screws is turned a tiny bit. Individually, this achieves almost nothing, but en masse it gives rise to language understanding.

The trick is that nobody has to label the training data by hand. The correct answer is already contained in the text itself, namely the word that actually follows there. Experts call this self-supervised learning. Only through this can such enormous amounts of data be used at all.

A common misconception: after pretraining, the model is not yet a helpful assistant. It continues texts instead of answering questions. Only in further steps does it learn to follow instructions and avoid failures. This post-processing is called fine-tuning, and for it comparatively small, carefully curated sets of examples are used.

Pretraining in news and products

Reports about new models are almost always about pretraining. What gets mentioned then is the number of parameters, the amount of text chunks processed, and the number of graphics chips deployed. When a corporation announces a new data center, it’s usually preparation for the next round of pretraining.

The industry’s legal disputes also revolve around this phase. Newspapers, publishers, and artists are suing because their works ended up in the training data without permission. At the same time, providers are signing licensing agreements with media companies. This is relevant for investors because such proceedings can change the costs of future models.

In everyday life, one tends to encounter the term rather indirectly. If a model description states a date like “knowledge as of mid-2024,” that refers to the end of the pretraining data. And if a company says it has “adapted” an open model, it did not carry out the expensive pretraining itself, but only the final fine-tuning work.