Distillation
Distillation is a technique in which a large, expensive AI system teaches a smaller one to give similar answers. The result is a compact model that runs faster, needs less computing power, and still retains much of the original's capability.
An AI model is a program that has learned to solve tasks from a huge number of examples. The most capable of these programs are enormous. They need expensive specialized computers and a lot of electricity just to respond at all. In distillation, therefore, a small program is built and made to imitate the answers of the large one. The large one is called the teacher, the small one the student. In the end, the student handles many tasks almost as well as the teacher, but costs only a fraction to operate.
Why small models save so much money
Every answer from an AI model costs computing time, and computing time costs money. For a service with millions of users, this adds up daily. A distilled model can be ten times smaller than its teacher. Power consumption and server costs drop accordingly, often by a whole order of magnitude.
Then there’s the speed. Small models answer faster because they have less to compute. You notice this immediately with a chatbot. And only a small model can fit on a phone or in a car in the first place. There’s no data center there to help out.
For companies, this is strategically important. Training a top-tier model costs hundreds of millions of dollars. A distilled model based on it is comparatively cheap. This is why there is also dispute over whether one may use other companies' models as teachers. Several providers explicitly forbid this in their terms of use.
What the student picks up from the teacher
The same question is sent to both teacher and student, and the answers are compared. If the student’s answer deviates, its internal settings are slightly adjusted. These values are called parameters and are the dials that determine the model’s behavior. After millions of such rounds, the answers converge closely.
The trick lies in exactly what is being imitated. A language model doesn’t simply decide on a single next word. It assigns probabilities to all possible words. The teacher passes on this entire distribution, not just its favorite. Within it lies the information about which alternatives would have been almost as good.
You can picture this like a teacher who doesn’t just state the solution but also says which mistakes would be plausible. The student learns more from this than from a bare model answer. This is precisely why distillation works better than simply training the small model from scratch on the same data. Some methods go even further and have the teacher also explain its intermediate steps.
Distilled models in products and headlines
Almost every major provider sells a smaller, cheaper variant alongside its top-tier model. Name additions like Mini, Flash, Haiku, or Nano usually indicate this. Such versions often arise through distillation from the large sibling model. In the price list, they frequently cost less than a tenth per request.
This technology is also found in phones. Offline translation, speech recognition, and autocorrect run on small models directly on the device. This is faster and protects data, since nothing is sent to a server. Assistance systems in cars or cameras that recognize objects work similarly.
Distillation made major headlines in early 2025. OpenAI accused the Chinese provider DeepSeek of having misused its own models as teachers. Such a thing is hard to prove, because you can’t tell from the student who it learned from. It’s also important to draw a distinction here: distillation builds a new, smaller model. Quantization, by contrast, merely stores the numbers of an existing model more coarsely and leaves its size essentially unchanged.