Knowledge Distillation

Programs like ChatGPT consist of enormous computational models that have learned from example texts. Such models can do astonishingly much, but require expensive specialized computers to do so. Knowledge distillation is a way of transferring this ability to a significantly smaller program. The large model is called the teacher, the small one the student. The student is not set loose on the world anew, but instead learns specifically to imitate the teacher’s answers. The result is a program that works almost as well but costs only a fraction of the computing power.

Why companies want small models

Operating a large language model costs money with every single request. If millions of people ask questions daily, this adds up to substantial amounts. A distilled model can often perform the same task ten times more cheaply. That is why behind many affordable AI services lies a small model that has learned from a large one.

Speed is another factor. Small models respond faster because they have to compute less. For a translation app or autocomplete feature, a delay of half a second is already disruptive. In addition, small models fit on devices you hold in your hand. They then run directly on the smartphone without sending data to a remote server. This is also an advantage for data privacy.

One side effect is controversial: you can distill other companies' models by bombarding them with masses of questions and using the answers for training. Providers explicitly forbid this in their terms of use. Whether such prohibitions hold up legally has not yet been clarified. This is precisely why there have been several public disputes between AI companies in recent years.

What the student picks up from the teacher

An AI model rarely outputs just a single answer. Internally, it assigns probabilities to many possible answers. For an image of a cat, it might say: 80 percent cat, 15 percent lynx, 1 percent dog. This graduated assessment is the actual treasure. It reveals which things look similar to the model.

In normal training, a model is only presented with the hard truth: cat, done. In distillation, the student instead receives the teacher’s full probability distribution. This means it learns not only the correct answer, but also how confident one is allowed to be. Experts call these graduated values soft targets. They contain more information per example and make the student’s training more efficient.

With language models, things usually work a bit differently. There, the teacher first generates large quantities of example answers, often including spelled-out intermediate steps. These texts are then used as training material for the student. It is important to draw the line against mere model shrinking: with quantization, you run the same model just less precisely, whereas distillation produces its own, newly trained model.

Recognizing distilled models in everyday life

Many providers release their models in several sizes. Name additions like Mini, Flash, Lite, or Haiku almost always indicate a smaller model that has benefited from a larger variant. In price lists, these versions appear with significantly lower costs per request. Anyone building AI into their own app often chooses them deliberately for simple tasks.

You also encounter the technique constantly on your own device. Speech recognition on your phone, keyboard text suggestions, and camera image enhancement all use small models. At full size, they would be far too power-hungry for a battery. Without distillation and similar techniques, every one of these functions would have to run over the internet.

A common misconception is that a distilled model is simply a copy of the large one. That is not true. The student has less capacity and therefore loses capabilities, usually in rare specialized knowledge and long chains of reasoning. This barely shows up on standard tasks, but is noticeable on tricky questions. That is why many services combine both approaches: the small model answers first, and the large one steps in for difficult requests.