Knowledge Distillation
Knowledge distillation is a training method in which a large, expensive AI model teaches a smaller one to behave as similarly as possible. The result is a lean model that answers almost as well but requires significantly less computing power.
Programs like ChatGPT are based on models: huge computational rules that have learned from many examples. The best of these models are very large and require expensive specialized computers to run. Knowledge distillation is a way to transfer this capability to a smaller model. To do this, the large model is made to solve many tasks, and the small model is trained to produce the same outputs. In this setup, the large model is called the teacher, the small one the student. The end result is a model that is often only a tenth the size yet retains most of the performance.
Why small models are suddenly competitive
Using an AI model costs electricity and computing time with every single request. With millions of users, this adds up to one of the biggest cost blocks for a provider. A smaller model directly reduces these costs, often by a whole order of magnitude. This is precisely why distillation is so economically attractive for companies.
There is also the matter of where the model runs. Large models run in data centers, small ones fit on a laptop or a smartphone. This means data stays on the device, and the response arrives without an internet connection. For applications in clinics, government agencies, or cars, this is often more important than the last few percentage points of quality.
It’s important to distinguish this from related methods. In quantization, the numbers of a model are stored more coarsely, but the model itself remains the same. In distillation, on the other hand, a new, independently trained model is created. Both can be combined.
What the teacher reveals to the student besides the answer
A language model doesn’t simply output a word. It calculates a probability for each of many possible words. After the sentence beginning “The dog barks loudly in the”, “garden” might get 60 percent, “yard” 25 percent, and “room” 5 percent. This complete distribution is the real treasure in distillation.
If a model is trained only with finished texts, it learns exclusively: here goes “garden”. If, on the other hand, it receives the teacher’s distribution, it additionally learns which alternatives were also plausible. This extra information is called soft knowledge. It packs a lot of experience into few training examples and makes learning more efficient.
In practice, this often works the other way around from what the name suggests. The teacher is made to solve a very large number of tasks, often with detailed intermediate steps. These answers are collected and form the training dataset for the student. A typical misconception is that the student could surpass the teacher this way. It can pick up the teacher’s errors and biases along with everything else, but it can rarely develop abilities the teacher did not have.
Distillation in products and in the headlines
Almost every major provider now sells several sizes of the same model. Names with additions like “mini”, “flash”, “lite”, or “small” usually indicate distilled variants. Anyone using a programming interface pays only a fraction of the price for these versions. The AI features built directly into newer smartphones are also based on such shrunk-down models.
In the news, the term also appears as a point of contention. Providers frequently prohibit, in their terms of use, training a competing model using the outputs of their model. In early 2025, there were public accusations exactly about this against the Chinese lab DeepSeek. Proving it is difficult, because a finished model rarely reveals where its training data came from.
For companies, distillation is also a standard route to specialized models. A strong general-purpose model is taken as the teacher and distilled for a single task, such as sorting customer inquiries. The result is often on par for that one task, but permanently cheap to run.