Foundation Model

In the past, a separate computer program was built for each task, one that learned from examples: one for speech recognition, one for translation, one for detecting cats in photos. Each of these programs only ever learned its one task and could do nothing else. A foundation model turns this around. A single, very large model is first made to work through gigantic amounts of text and images without being given a specific task. In doing so, it learns broad general knowledge about language, relationships, and patterns in the world. Only afterward is it aligned to concrete tasks, and for that, relatively few additional examples suffice. The term was coined in 2021 by researchers at Stanford University and has since become established across the industry.

Why one model is enough for hundreds of applications

The economic appeal lies in reuse. The expensive, months-long base training happens once and often costs hundreds of millions of euros. After that, the same model can power a customer service chat, a search engine, and a programming tool. The costs of the base training are thus spread across a great many products.

This also explains why only a few corporations build such models themselves. Doing so requires tens of thousands of specialized computing chips and access to enormous amounts of data. Names like OpenAI, Google, Meta, Anthropic, or Mistral therefore keep appearing in the news again and again. All other companies are customers of these providers and build their products on top.

It is precisely this dependency that makes experts nervous. If a foundation model contains an error or a bias, every application built on top of it inherits the problem. A single faulty building block then has an effect across thousands of products at the same time. This is referred to as a concentration risk.

Pretraining, fine-tuning, deployment

In the first phase, pretraining, the model predicts which word comes next, billions of times over. Nobody has to label the data for this by hand, since the text itself supplies the correct answer. With every mistake, the model’s internal adjustment knobs are minimally recalibrated. These knobs are called parameters, and large models have hundreds of billions of them.

The second phase is fine-tuning. The finished model is trained further using a comparatively small, carefully selected set of data. This is how it learns, for example, to respond politely, to summarize medical letters, or to get legal terminology right. This step takes hours or days rather than months.

A comparison helps with understanding: pretraining corresponds to general schooling, fine-tuning to on-the-job training. It is important to distinguish this from the term large language model. A large language model works only with text and is thus a special case. Foundation models can also process images, audio, or protein structures.

Foundation models in products and headlines

Anyone using ChatGPT, Gemini, or Claude is talking to the fine-tuned version of a foundation model. The image generators that produce illustrations on command also belong to this category. In medicine, such models scan X-ray images; in research, they predict how proteins fold. The pattern is always the same: a broadly trained base model, with a narrow adaptation layered on top.

In stock market news, you mostly encounter the term in connection with money. When a provider raises billions, the money goes predominantly into data centers for the next round of base training. The European AI Act also uses a related expression: it speaks of AI models with general purpose and imposes additional requirements on especially powerful models.

A common misconception is the idea that a foundation model knows everything. Its knowledge ends on the day the training data was collected. Current information has to be supplied to it along with the query, for instance via a connected search function. The model is a broad foundation, not an all-knowing reference work.