Model Distillation

Large AI systems like those behind chatbots consist of enormous quantities of numerical values that were set during training. The more of these values a system has, the more computing power each individual answer costs. Model Distillation is a way to reduce these costs. A large, highly capable system is used to train a smaller one. The large one is called the teacher, the small one the student. The student adopts as much as possible of the teacher’s behavior while remaining small and fast. The name comes from chemistry: as with distilling, what remains should be the essential part.

Why small models are suddenly enough

Running a large AI system is expensive. Every request occupies special graphics chips in a data center, and those cost money and electricity. With millions of users, this adds up to enormous sums. A distilled model can be five or ten times smaller while performing almost as well on everyday tasks.

Then there’s speed. Small models respond faster because less computation is needed. For a translator in an app or a search suggestion feature, waiting time is a real problem. Some distilled models are so small that they run directly on a phone or in a car. In that case, the data stays on the device and doesn’t need to be sent over the internet.

One side effect is economically explosive. Anyone with access to a powerful model can transfer some of its capabilities into their own, cheaper model. Providers often explicitly prohibit this in their terms of use. The accusation that a competitor has distilled another company’s models keeps coming up regularly in the industry.

What the teacher reveals to the student

The naive approach would be to feed the student the teacher’s finished answers. This works, but it wastes information. That’s because a language model never decides in a hard, binary way — instead, it assigns a probability to every possible next word. It is precisely this distribution that is valuable during distillation.

An example: asked about the capital of Australia, the teacher answers with 90 percent Canberra, 7 percent Sydney, and 3 percent Melbourne. From this, the student doesn’t just learn the correct answer. It also learns which mistakes are plausible and which would be completely far-fetched. Experts call this intermediate information soft targets, as opposed to a single hard correct solution.

With modern language models, a second variant comes into play. The teacher generates large quantities of example tasks along with detailed solution paths. The student then trains on this self-generated material. This is especially effective for math and programming, because the student imitates the step-by-step approach. There are limits, though: a student rarely becomes better than its teacher, and the teacher’s mistakes get passed along too.

Distilled models in products and headlines

Nearly every major provider now sells several sizes of the same model. The smaller variants with additions like mini, flash, or nano are often distilled or otherwise shrunk down. They cost a fraction per request and are the usual choice for simple tasks. Developers combine both approaches: small models for routine work, the large one only for difficult cases.

The technology is also embedded in phones. Speech recognition, camera effects, or text suggestions often run on shrunken models directly on the device. In the news, distillation became a hot topic especially in early 2025, when the Chinese company DeepSeek introduced surprisingly cheap models. American providers voiced suspicions at the time that their systems had been used as teachers.

A common misconception: distillation is not the same as quantization. With quantization, the model stays the same size, only its numbers are stored more coarsely. With distillation, on the other hand, a new, smaller model is created. In practice, both are often applied one after the other.