Logit Distillation

Large AI models are expensive to run, small ones are cheap but often weaker. Logit Distillation is a way to let a small model learn from a large one. The large model is called the teacher, the small one the student. The trick lies in what the student gets to see. It doesn’t just learn which answer the teacher gives, but how confident the teacher was about every possible answer. These raw confidence values, which a model internally calculates for each possible answer, are called logits — hence the name.

Why the teacher’s doubts are valuable

Normal training only shows the model the correct answer. The image shows a dog — full stop, no further information is given. This is called a hard target. The teacher, on the other hand, delivers a soft target: 70 percent dog, 25 percent wolf, 4 percent cat, a tiny remainder for everything else. This distribution contains a statement missing from the hard target: dogs and wolves look similar, cats are further away.

Researchers call this the teacher’s dark knowledge. It comes free of charge and describes how the large model has sorted the world. The student doesn’t have to laboriously work out this order itself from raw data. This is why it learns with fewer examples and less computing time.

Economically, this is decisive. A model that is ten times smaller costs roughly ten times less per request. It also fits into devices the large one would never fit into. Many of the compact models that companies release for free have been created this way from a larger model.

Temperature, blur filter and loss function

Logits are initially just numbers without a fixed scale. To turn them into probabilities, a calculation formula called softmax is applied. It compresses the values so that they add up to 100 percent. The problem: large models are often very decisive. They give the correct answer 99 percent and almost nothing to everything else. Then hardly anything remains of the interesting additional knowledge.

This is why a control called temperature is used. A high temperature pulls the distribution apart, like a blur filter over an image. From 99 to 1, it might then become 60 to 40. The ranking stays the same, but the fine differences become visible. Teacher and student are viewed at the same temperature so the comparison is fair.

During training, a loss function measures how much the two distributions deviate from each other. The student is adjusted until the gap is small. Usually both are mixed: part of the training aims at the teacher, part at the actual correct answer. It’s important to distinguish this from related methods. In Feature Distillation, the student also copies the teacher’s internal intermediate results. Logit Distillation only looks at the output and is therefore simpler — the models don’t need to resemble each other internally.

Small models on the phone and disputes over teacher access

When a voice assistant runs directly on a smartphone without an internet connection, a distilled model is almost always behind it. The same applies to translation functions in the browser or cameras that recognize objects offline. The smaller variants of large model families, often recognizable by abbreviations like mini, small or flash, are also frequently created this way.

In the news, the term usually appears in connection with disputes. Providers prohibit in their terms of use using the responses of their model to train a competing model. This exact accusation was raised in 2025, when OpenAI alleged that the Chinese provider DeepSeek had used outputs from GPT models for distillation. A common misconception here is that distillation always requires actual logits. Through a public interface, you usually don’t get these at all — then the student only learns from the teacher’s finished texts, which appears weaker but also works.