Knowledge Distillation
Knowledge Distillation is a training method in which a large, expensive AI model teaches a smaller one to behave similarly. The small model afterward needs less computing power and often even fits on a smartphone.
Programs that write texts or recognize images consist of billions of numerical values. The more such values a program has, the more it can usually do – but the more expensive each individual answer also becomes. Knowledge Distillation is a way to ease this problem. A large, capable program is given a great many tasks, and a small program is trained to give the same answers. The large one is called the teacher, the small one the student. The result is a program that is significantly smaller but in many cases behaves almost as well as its model.
Why small models are suddenly competitive
The most important reason is money. Every request to a very large model costs computing time in a data center. With millions of users per day, this adds up to enormous amounts. A distilled model at a tenth of the size can often handle the same task for a fraction of the cost.
The second reason is the location of the computation. Large models necessarily run on servers because they require a lot of memory. Small models fit on a phone, a car, or a smartwatch. There, the data stays on the device, and it also works without an internet connection. The translation feature that works offline is typically a distilled model.
Then there is speed. A smaller model responds faster because it goes through fewer computational steps. With voice assistants or search suggestions, half a second decides whether something feels smooth. That’s why many providers don’t actually use their strongest model in the background, but a distilled variant of it.
What the teacher reveals besides the correct answer
The decisive trick lies in what the student imitates. One could simply present it with the correct solutions, as in normal training. In distillation, however, it gets to see the teacher’s complete assessment. An image recognition model doesn’t just say “cat,” but for example: 80 percent cat, 15 percent lynx, 0.01 percent car.
These secondary probabilities are valuable. They show the student which things resemble each other and which have nothing to do with one another at all. A textbook only states the solution; a good teacher additionally explains which confusion is likely. It is precisely this extra knowledge that makes the training more efficient than learning from raw data.
With language models it works similarly, but often via texts. The large model generates thousands of example answers, some with spelled-out intermediate steps. The small model trains on this self-generated material. The limit is important: the student rarely becomes better than the teacher. Distillation makes knowledge available more cheaply, it does not create new knowledge.
Mini versions, licensing disputes, and replication
In product names, the method almost always appears hidden. When a provider offers variants such as “Mini,” “Flash,” “Lite,” or “Nano,” distillation is frequently behind it. Freely available models sometimes also carry a “distill” in their name. They are designed to run on ordinary hardware instead of expensive specialized chips.
In business news, it’s mostly about the dispute over this. The terms of use of many providers prohibit using their models as a teacher for competing products. Precisely this accusation was in the air in 2025, when the Chinese company DeepSeek presented a very cheap model. The suspicion: it had been trained on the answers of other models.
The case shows why the topic interests investors. If the capabilities of a top model can be replicated for comparatively little money, the market leader’s lead shrinks. A technical detail from training thus becomes a question about business models and the valuations of entire companies.