Self-Speculation

Programs like ChatGPT write their answer piece by piece. They compute one word fragment, append it, and start over. Each of these steps costs time, because the entire, huge program has to be run through once. Self-Speculation reverses this process: first, a fast, simplified version of the same program guesses several upcoming word fragments at once. Then the full version checks all the guesses in a single computation pass. If they’re correct, they’re accepted; if not, computation restarts from the first incorrect point.

Why waiting time is more expensive than computing power

Anyone working with a chatbot notices delays immediately. A model that outputs three words per second feels sluggish. The same model at twenty words per second feels smooth. The quality doesn’t change at all, only the feel of using it. That’s exactly why providers invest so much effort into such speedups.

There’s also a technical reason. Modern graphics chips can perform an enormous number of calculations simultaneously. But with classic word-by-word generation, they sit idle most of the time. The bottleneck isn’t the computing itself, but reloading the model weights from memory. If the chip checks five guessed word fragments at once, it only has to load the weights once instead of five times.

For providers, this translates directly into hard cash. The same hardware can suddenly serve significantly more users in parallel. In practice, developers report speedups ranging from 1.5x to 3x depending on the task. For highly predictable text, such as program code, the gain is larger than for creative text.

Guessing and checking within the same model

The older relative of this method is called speculative decoding. There, you need two separate models: a small one for guessing and a large one for checking. The small model has to be trained separately and kept in memory. Self-Speculation avoids exactly this extra overhead. The prefix “Self” means: the model guesses ahead of itself.

Technically, there are several ways to achieve this. One common approach skips part of the model’s layers while guessing. Large language models are built from dozens of stacked processing stages, which one can picture as workstations along an assembly line. If you only use every second station, you get an imprecise but very fast preview model. Another approach attaches small additional heads to the model that predict several word fragments simultaneously.

The crucial part is the checking step. The full model computes once over the entire guessed sequence of words and compares it with what it would have chosen itself. All suggestions up to the first deviation are accepted, the rest discarded. Mathematically, it is guaranteed that the end result is exactly the same text as without the trick. So Self-Speculation doesn’t make the model dumber or less accurate, just faster.

Where the trick runs in the background

Self-Speculation is never directly visible. It’s built into the software that runs models, for example in libraries like vLLM or in the services of large providers. When a chatbot update suddenly responds noticeably faster without a new model, such a speedup is often behind it. In trade news, the term usually appears alongside keywords like Medusa, EAGLE, or layer skipping.

The method is especially interesting for AI on one’s own device. On a laptop or smartphone, memory is scarce, and a second guessing model would be a nuisance. Self-Speculation gets by with the one existing model. That’s why the approach is increasingly found in tools used to run language models locally.

A common misconception: that Self-Speculation has something to do with “thinking” or self-correction. That’s not true. The model doesn’t check the content of its statements for truth. It only checks whether the quickly guessed word fragments match its own normal output. False claims remain just as false, they simply appear on screen more quickly.