Base Model

A base model is the first, unfinished stage of a computer program that processes language. Over a months-long computing process, it has read enormous amounts of text, such as books, web pages, and program code. In doing so, it learned only a single task: predicting which word comes next. That is exactly what it can do very well afterward, and nothing more at first. So anyone who asks it a question often doesn’t get an answer, but rather a plausible continuation of the text. Asked “What is the capital of France?”, it might simply write three more exam questions to go along with it.

The raw version from which all chatbots emerge

Almost every well-known AI product starts out as a base model. The expensive part of the work lies exactly here: the initial training costs tens or hundreds of millions, depending on size, and requires thousands of specialized computing chips. Everything that comes afterward is cheap by comparison. That’s why the base model is the actual asset of an AI company.

Many different products can be derived from a single raw version. A provider can turn it into a chat assistant, a coding aid, or a model for medical texts. There’s no need to start from scratch each time. That explains why several variants of a model family often appear at once.

It’s important to distinguish this from the so-called instruct model. That’s the same model after post-training, i.e., the version that follows instructions. When a company makes its model freely available for download, it sometimes publishes both versions. Researchers then tend to prefer the base version, since it hasn’t yet been pushed in any particular direction.

From word-guessing to the raw version

Training works like a gigantic fill-in-the-blank game. The program is given a text excerpt and has to guess the next word. It is then shown the correct word. If it was wrong, its internal settings are adjusted slightly. This process repeats itself trillions of times.

These settings are called parameters. A large model has hundreds of billions of them. No one sets them by hand; they emerge purely from the data. Along the way, a great deal of world knowledge arises as well, since predicting the next sentence in a chemistry book requires having picked up something about chemistry.

After that come the fine-tuning steps. In so-called instruction tuning, the model is trained using example pairs of instruction and matching response. Afterward, humans rate different responses, and the model learns which ones are preferred. Only through this does a text-continuer become a conversation partner that answers questions and declines sensitive requests.

Where base models show up in the news

In everyday life, one rarely encounters base models directly. Anyone using ChatGPT or a similar service is always talking to the post-trained version. Base models become visible mainly on platforms like Hugging Face, where developers download models. There, the name often includes an addition like “base” or “instruct”.

The term comes up in business news when reporting on training costs and chip demand. Political debates also revolve around this stage: the dispute over copyright concerns exactly the amounts of text that flow into the initial training. And when a company calls a model “open source,” it usually means that the fully trained parameters are freely available.

A common misconception is that a base model is simply a smaller or worse model. That’s not true. It can be just as large and knowledgeable as the final version — it’s simply untamed. Companies wanting to build their own specialized AI therefore deliberately start here and layer their own fine-tuning on top.