Speculative Execution

A computer often has to wait. It reaches a point in the program where there are two possible continuations, and the decision about which one applies hasn’t been fully computed yet. Instead of sitting idle, it guesses how the decision will turn out and starts computing ahead in that direction. This is exactly what is called speculative execution. If the guess was correct, the work is already done and time has been saved. If it was wrong, the intermediate results are thrown away, and the computer starts over from the branch point.

Why waiting is more expensive than guessing

Modern processors work like an assembly line. An instruction isn’t completed in one go but passes through several stations one after another. For the line to keep moving, fresh material must constantly arrive. As soon as a decision is pending, this supply is missing, and the line runs empty.

Such a gap doesn’t cost just a few percent — it can cut speed in half. Branches are also very common: typical program code contains one roughly every five to ten instructions. Without speculation, a modern processor would therefore be noticeably slower, even though its components are just as fast. This technique is one of the reasons computing power has risen over decades without clock frequency increasing at the same rate.

The benefit depends entirely on how good the guessing is. With a hit rate of 95 percent, the effort pays off significantly. At 50 percent, half the work would be wasted, and the advantage would be gone.

Guess, compute, discard

There is a dedicated circuit in the processor for guessing: branch prediction. It remembers how a particular point in the program has turned out in the past. A loop that runs a thousand times ends in 999 out of 1000 cases with “continue.” The prediction recognizes this regularity and afterward is almost always right.

While speculation is happening, the results must not cause any permanent effects. They land in a temporary buffer and are only officially committed once the decision is settled. If the guess turns out to be wrong, the processor clears this buffer. From the outside, it looks as if nothing had ever happened.

However, this rollback isn’t entirely without a trace. The Spectre and Meltdown security vulnerabilities, published in 2018, exploited exactly that. Speculatively loaded data left traces in the cache, from which attackers could draw conclusions about other people’s data. The countermeasures at the time measurably cost performance.

From processor to chatbot

A language model generates text word by word. Each new word requires a complete pass through the large model, and that is the bottleneck. In speculative decoding, a small, fast model is therefore allowed to guess first. It writes five or ten words ahead.

The large model then checks this proposal in a single pass. Everything it would have written the same way itself is accepted. From the first deviation onward, the rest is discarded. The result is word-for-word identical to what the large model would have produced on its own — just two to three times faster. A common misconception is that this makes the answer worse or less accurate; that is not the case.

In practice, this technique is encountered constantly without being visible. It’s built into the chips of laptops and smartphones and into the servers behind every chatbot. In corporate announcements, it usually appears as the reason given for lower operating costs or shorter response times.