Distillation (AI)
Distillation is a process in which a large, expensive AI program teaches a smaller one to behave as similarly as possible. Afterwards, the small program needs far less computing power and often even fits on a phone.
Large computer programs that write texts or recognize images need a great deal of computing power. That’s why they usually run in huge data centers and are expensive to operate. Distillation is a trick to change that. The large program is made to solve a great many tasks, and its answers are recorded. A smaller program is then trained on exactly these answers, meaning it is adapted to imitate them. The end result is a program that needs much less memory and electricity, but performs almost as well on many tasks.
The price per answer
Every answer given by an AI system costs computing time, and computing time costs money. A provider with millions of users pays these costs anew every single day. A distilled model can easily be ten times cheaper than the original. This is exactly why distillation is so attractive for companies.
Then there’s speed. Small models respond faster because they have less to compute. For a chat, a search function, or a translation in the browser, this is crucial. Nobody likes waiting ten seconds for a single sentence.
The third reason is where the computation happens. A small model runs directly on a phone or laptop, without a detour through the internet. The data then stays on the device, which is a real advantage for privacy. Something like this even works offline.
Teacher, student, and the soft answers
Experts call the large model the teacher and the small one the student. The teacher is given thousands of example tasks and produces answers to them. These pairs of task and answer form the training material for the student. The student is adjusted until its outputs closely resemble those of the teacher.
One detail matters here: the student doesn’t just learn the finished answer. It also sees how likely the teacher considered the alternatives to be. If the teacher guesses 70 percent wolf and 25 percent husky for an image, that contains the information that these two are similar. This is referred to as soft answers, and they are more instructive than a simple right or wrong.
A common misconception is that the small model is simply a trimmed-down copy of the large one. That’s not true. The student often has a completely different internal structure and starts out as an empty model. Only the behavior is transferred, not the construction. That’s why the student usually remains weaker in edge cases: whatever the teacher never had to demonstrate in the training tasks, the student doesn’t learn either.
Mini models in products and the dispute over training data
Nearly every major provider now sells several sizes of the same system. Names like Mini, Flash, Haiku, or Nano indicate small variants that have often been distilled. In practice, an app chooses the size to match the task. For a simple summary it uses the small model, for a tricky analysis the large one.
The technology is also embedded in devices. The translation and dictation functions of modern smartphones use heavily shrunken models. The same goes for voice assistants in cars, which are meant to respond even in areas without signal. Without distillation, such features would hardly be affordable.
In the news, the term also comes up in a contentious topic. It is in fact possible to use other providers' models as teachers by querying them en masse and collecting the answers. Many providers' terms of service explicitly forbid this. When the Chinese model DeepSeek R1 appeared in early 2025, exactly this accusation was raised. Such cases are hard to prove, but they shape the debate over fair competition in AI.