Draft Model

Programs like ChatGPT write their answers word by word. For every single word, the program has to run through its calculations once, and with very large programs this noticeably costs time. A draft model is a second, much smaller program of the same kind that can also write the same text – just faster and somewhat less precisely. It delivers a quick draft, which the large program then proofreads. If the draft fits, it is accepted. If it doesn’t fit, the large program corrects it. The English word “draft” means exactly that: a rough version.

Why answers appear faster as a result

The time between question and answer plays a role in whether a chatbot feels pleasant to use. Users clearly notice delays of half a second. Depending on the task, a draft model can speed up output by roughly two to three times. The text produced remains the same one the large model would have produced on its own. That is exactly the crucial point: you gain speed without trading away quality.

For operators, money matters too. Every answer consumes computing time on expensive graphics cards, and that computing time is billed per second. If a large model can check several words at once instead of generating them one by one, the cost per answer drops. With millions of requests per day, this adds up considerably.

One distinction is important: a draft model does not make the AI smarter. It changes nothing about what the large model knows or how well it solves tasks. It only changes how quickly the same answer is produced. Anyone who wants better answers needs a better main model, not a draft model.

Guessing and proofreading as a duo

The procedure is called speculative decoding, because the small model guesses ahead on a hunch. For example, it writes four or five words in one go. Afterwards, the large model is presented with the draft and checks all the words at once. This simultaneous checking is the trick: for a graphics card, evaluating five words costs barely more time than generating one.

When checking, the large model compares word by word whether it would have chosen the same one itself. All words up to the first deviation are accepted. From the deviation onward, the rest of the draft is discarded, and the large model sets the next word itself. Then the round starts over from the beginning. Because the check is strict, the result can never be worse than without a draft model.

A comparison helps: an assistant drafts an email in advance, and the boss reads it through. If everything is correct, it goes out. If the third sentence contains nonsense, it gets rewritten from that point on. This is only worthwhile if the assistant is right most of the time. That’s why a small model from the same family is usually chosen as the draft model, for instance a model with one billion parameters serving as a helper for one with seventy billion. If it guesses wrong too often, the constant discarding costs more time than the trick saves.

Where the technology is already in use

You never see draft models directly, because they operate in the background of the systems. Large chat services and coding aids such as code autocompletion use this procedure to make answers appear more fluid. It is also interesting on laptops and smartphones, since computing power is limited there. Common software for running AI models now comes with this feature built in.

In technical news, the term usually appears together with “speculative decoding”, “latency”, or “tokens per second”. Latency refers to the wait time until the first word, tokens per second to the output speed afterward. When a provider announces that its service responds twice as fast with the same model, this technique is often exactly what’s behind it. Related approaches save computing time differently, for instance by storing numbers within the model more coarsely.