On-Policy Distillation
On-policy distillation is a training method in which a small AI model generates its own responses and a large model evaluates these responses step by step. This way, the small model learns exactly where it actually makes mistakes — rather than being trained past someone else's sample solutions.
Large language programs like ChatGPT are expensive to run because they require enormous computing power. That’s why efforts are made to transfer their capabilities to a smaller, more economical program. This transfer is called distillation: a large teacher passes its knowledge on to a small student. In on-policy distillation, this happens according to a particular pattern. The student first writes its own answers, and the teacher then states, for each individual word, how well it would have fit. The English term “on-policy” means exactly that: practice happens on what the student produces on its own.
Why sample solutions alone aren’t enough
In the classic approach, the small model is presented with finished texts from the large model and is meant to imitate them. This works, but it has a design flaw. The small model only practices on sentences it would never have written that way itself. In real-world use, it then quickly ends up in situations that never occurred during training.
Experts call this problem error accumulation. A first small slip pushes the model onto an unfamiliar path. It has never practiced there, so the second error becomes bigger, the third bigger still. With long chains of reasoning or multi-step arguments, an answer can collapse entirely as a result.
A comparison from sports captures it well. You can watch videos of professionals as much as you like. You only get better once someone stands beside you during your own game and corrects your own mistakes. On-policy distillation makes exactly this shift.
What the teacher evaluates at every word
A language model writes text word by word. At each step, it calculates a probability for thousands of possible continuations. This distribution is more than just the decision for one word. It shows how confident the model is and which alternatives it considered.
During training, the student first generates a complete answer. The teacher then reads this answer and calculates its own probability distribution for each position. The training software compares both distributions and measures the distance between them. The student is then adjusted so that its behavior comes closer to that of the teacher.
This is significantly more information than a single grade at the end. In the related method of reinforcement learning, there is often only one signal for the entire answer: right or wrong. Here, by contrast, the student receives feedback on every single word. This is precisely why on-policy distillation requires fewer training runs and is considered comparatively compute-efficient.
Small models in products and headlines
The term comes up when providers introduce compact variants of their models. Typical name suffixes are “mini,” “flash,” or “small.” Such versions are meant to answer almost as well as the large original, but at a fraction of the cost. Distillation methods are a central building block for achieving this.
This becomes especially important for models meant to run on a phone or in a car. There is no data center in the background there. Companies also use the technique to tailor a general model to their specific field, such as legal texts or customer inquiries.
A common misconception is that a distilled model is a shrunk-down copy of the teacher. That’s not true. It mainly adopts the response behavior in a specific domain, not the entire body of knowledge. Moreover, the teacher remains the ceiling: through distillation, a student practically never becomes better than the model it learns from.