Decoding

A program like ChatGPT doesn’t write text in one go. It works piece by piece, always appending just one small text unit, usually a word or part of a word. And it never outputs a single fixed answer. Instead, for thousands of possible continuations it delivers a probability for each: “capital” might fit here with 40 percent, “largest city” with 12 percent, “currywurst” with 0.001 percent. Decoding is the step that turns this long probability list into a concrete decision. You can picture it as a selection mechanism that sits behind the actual model and puts the word in its mouth.

Why the same chatbot answers differently twice

Many people believe the quality of an answer depends solely on the model. That’s only half the truth. The same model can appear completely different with different decoding settings. Once boring and repetitive, once original, once completely incoherent.

This is also tied to a practical detail that irritates many users. If you ask a chatbot the same question twice, you often get two different answers. That’s not a bug and not a mood swing. The selection process contains an element of randomness, which is why the decision doesn’t turn out the same every time. Anyone who needs reproducible results, for example in a software evaluation, deliberately switches off this randomness.

For companies, this is a cost factor. Some decoding methods check multiple text variants in parallel and thereby compute a multiple of the workload. Others are frugal but produce flatter texts. You have to choose between quality, speed, and computing cost.

From greedy search to temperature

The simplest method is called greedy decoding, meaning greedy selection. It always takes the most probable continuation. That sounds sensible, but it often leads into dead ends and endless loops. The text then repeats itself, because phrasings once begun reinforce themselves.

One remedy is beam search. Here the system pursues several sentence beginnings simultaneously, say five, and in the end keeps the overall best one. This helps with translations, where there is a fairly clear correct solution. With free writing, however, the results often seem smooth and soulless.

Modern chatbots therefore roll the dice in a controlled way. Two adjustment knobs keep coming up here. Temperature regulates how strongly improbable words are allowed to play along: low temperature means cautious and fact-close, high temperature means adventurous and experimental. Top-p cuts off the list and only allows the candidates that together make up about 90 percent of the probability. This leaves room for creativity without letting completely absurd words slip through. It’s important to draw the distinction: decoding does not invent any knowledge. It only selects from what the model already considers possible.

Temperature dials in apps and interfaces

Anyone using a programming interface from a provider like OpenAI, Google, or Mistral sets decoding directly themselves. There you’ll find fields like temperature or top_p. Developers set the temperature low for a customer-service bot so it stays factual. For an ad-copy tool, they turn it up. Comparable dials also exist in image generators and music AIs.

In the news, you usually encounter this topic indirectly. When it’s reported that a chatbot made up a source out of thin air, decoding plays a role. A high temperature makes such hallucinations more likely, meaning freely invented but convincingly sounding claims. It’s not the sole cause, but it is an amplifier.

A current trend is pushing the boundary even further. In so-called speculative decoding, a small, fast model proposes several words on a hunch. A large model checks the proposals in a single pass and discards whatever doesn’t fit. The result stays the same, but the answer arrives noticeably faster. Tricks like this are one reason why AI services respond more fluidly today than they did two years ago.