Distillation

Programs like ChatGPT answer questions because they have previously practiced on huge amounts of text. Such programs are called models. The most powerful of these are very large and need expensive specialized computers just to run at all. In distillation, one such large model is used to train a small one: the large one answers thousands of example questions, and the small one practices giving the same answers. The result is a leaner model that has taken on a large portion of the capabilities. The name is borrowed from distillation in chemistry, where the essential substance is extracted from a liquid.

Why small models suddenly became interesting

Large models are expensive to operate. Every single answer costs electricity and computing time on graphics chips that cost several tens of thousands of euros apiece. Anyone running a service with millions of users pays these costs anew every day. A distilled model can often handle the same task for a fraction of the price.

Then there’s the size. A very large model only fits in a data center, while a distilled one sometimes fits on a laptop or even a smartphone. Then the question never has to leave the device at all. That is faster and more favorable for data that nobody wants to send over the internet, such as medical records or internal company documents.

Distillation also shifts the balance of power within the industry. Anyone who has trained a top-tier model at enormous expense must expect a competitor to query it and extract a cheap knockoff model from it. That is exactly why many providers explicitly prohibit, in their terms of use, using their answers to train one’s own models.

Teacher and student in training

The two parties involved are usually called the teacher model and the student model. The student model is built smaller from the outset, with fewer internal adjustable parameters. Both are given the same tasks and their outputs are compared. The student is adjusted repeatedly until its answers closely resemble those of the teacher.

One detail is decisive here. A language model does not just output one word, but a probability for many possible words. The teacher thereby reveals how confident it is and which alternatives it considered almost as good. These nuances contain more information than a mere model answer. That is why a student model often learns faster from teacher answers than from ordinary training texts.

Still, the copy never becomes perfect. A distilled model is usually strong in the areas that appeared in the training questions, and weaker outside of them. The teacher’s errors are also learned along the way. And distillation is something different from quantization: there, the model is kept as is and only its numbers are stored more coarsely. In distillation, by contrast, an entirely new, smaller model is created.

Distilled models in products and headlines

Many providers sell their models in size tiers, often with names like Mini, Flash, or Lite. These cheaper variants are frequently distilled versions of the large model. For standard tasks such as summarizing or translating, they are usually sufficient. Only for complicated tasks does one then still fall back on the expensive top-tier model.

The technology is also built into phones. Offline automatic translation, keyboard suggestions, or voice control often run on distilled models. They need to be small enough to fit in memory and not drain the battery.

In the news, the term usually appears in two contexts. Either a company celebrates that its small model nearly matches the large one’s performance. Or there is a dispute because one company accuses another of secretly distilling its model. Such accusations are hard to prove, because one cannot tell from the finished student model where its training answers originally came from.