Decoding

A language model is a computer program that continues texts by writing on word by word. In doing so, it never names a single next word. Instead, it calculates a probability for each of thousands of possible words. After “The capital of France is”, “Paris” might get 95 percent, “a” 2 percent, and the rest is spread across everything else. From this list, however, exactly one word must be selected, otherwise no text is produced. This very selection is called decoding. It is not part of the learning process, but a rule that is imposed on the finished model from outside.

Why the same model answers differently twice

Many people believe an AI model has a fixed answer to a fixed question. That’s not true. If you ask the same question twice, you often get two different texts. The reason lies not in the model, but in the decoding. If it deliberately rolls the dice a little, the choice sometimes falls on the second most probable word. From that point on, the entire rest of the sentence heads in a different direction.

This setting is a genuine dial, not a random byproduct. Someone having a model write program code usually wants the most predictable variant. Someone looking for ideas for a short story wants variety. The same model can deliver both, depending on the decoding. That’s why it’s worth knowing the difference before dismissing a result as poor.

An important distinction is also this: decoding does not make a model smarter. It can only select from what the model already considers possible. Incorrect knowledge cannot be fixed with it. A poor setting can, however, make a good model look considerably worse.

From the mountain of probabilities to a single word

The simplest strategy is called Greedy Search. It always takes the most probable word. That sounds reasonable, but it often leads to boring and repetitive texts. The reason is that the best individual decision doesn’t always lead to the best overall sentence. Someone hiking who always takes the steepest path upward ends up on the next hill instead of the summit.

That’s why the system usually rolls the dice deliberately. With Top-k, the system chooses only from the k most probable words, say the best fifty. With Top-p, also called Nucleus Sampling, it collects words until their probabilities together add up to a share such as 90 percent. Everything below that gets discarded. This way, nonsensical stuff with a tiny probability can’t even be drawn in the first place.

Another dial is called temperature. A low temperature makes the probabilities more contrasted, making the model more predictable. A high temperature evens them out, giving rare words a genuine chance. At a temperature of zero, the greedy method reappears. Very high values, on the other hand, quickly tip over into incoherent rambling.

Where the dials show up in chatbots and interfaces

Anyone using ChatGPT or a similar service normally sees none of this. The provider has fixed the values, usually to a middle ground. Decoding becomes visible where developers address a model via a programming interface. There, fields like temperature or top_p appear, often with values between 0 and 2.

In product interfaces, the same principle hides behind buttons like “creative”, “balanced”, and “precise”. Even the “regenerate response” button only works because decoding contains randomness. Without this randomness, the same text would come out every time.

In news and trade articles, the term is usually encountered in connection with cost and quality. More elaborate methods like Beam Search check several sentence variants in parallel and select the best one at the end. This often delivers cleaner translations but costs more computing time. So-called speculative decoding also belongs here: a small, fast model proposes several words, and the large one merely checks them. The result stays the same, but the answer arrives faster.