Decoder

A decoder is the component of a language program that writes new text. It works piece by piece: it looks at what has been written so far and predicts what best comes next. It appends this next piece and starts over. In this way, word by word, a whole sentence emerges, then a whole paragraph. The direction of view is important: the decoder is only allowed to look backward, that is, at what has already been written, never forward. The name is somewhat misleading, since “decoding” sounds like decrypting. What is meant is rather: turning an internal representation back into readable text.

Why chatbots now consist only of decoders

Older translation programs had two halves. One half, the encoder, read in the source sentence completely and compressed its meaning into numbers. The other half, the decoder, wrote the sentence in the target language from that. This division makes sense when input and output are clearly separated.

With a chatbot, this is not the case. Question and answer are both simply text, placed one after the other. The question can therefore be treated directly as the beginning of a text that the decoder continues. This is exactly what the GPT models do. Such systems are called decoder-only, that is, decoder-only models.

The practical advantage is enormous. A decoder-only model only needs huge amounts of ordinary text to learn, not cleanly translated sentence pairs. As a result, almost the entire internet was suddenly available as training material. This one simplification is a major reason for the leap in quality since 2020.

One word piece after another

The decoder does not work with words, but with tokens. A token is a text building block, often a syllable or a short word. For each new token, the model calculates probabilities across its entire vocabulary. If the text so far reads “The capital of France is,” then “Paris” gets a very high value and “potato” gets a very low one.

Then a token is selected. Not always the most probable one, since that quickly sounds stiff and repetitive. Usually, one is drawn at random from the best candidates. How much randomness is allowed is controlled by a value called temperature. That is why you get slightly different answers to the same question twice.

During training, the view forward is technically blocked. This is called masking. Without it, the model could simply read the solution off the text while practicing and would learn nothing. This enforced one-way street distinguishes the decoder from the encoder, which is allowed to look at a sentence in both directions simultaneously.

Decoders in everyday life and in the news

Every time the answer appears bit by bit in a chat window, you’re watching a decoder at work. The text is not displayed slowly for effect. It really only comes into being at that moment, token by token. The autocomplete feature in search fields and email programs is also based on the same principle.

In technical reports, phrases like “a decoder-only transformer with 70 billion parameters” appear. This is not a special feature but the standard design: Llama, Gemini, Claude, and the GPT series all belong to this category. Encoder models like BERT, on the other hand, are found working in the background, for example in search engines or when texts are automatically sorted.

A common misconception: because the decoder uses probabilities, it is supposedly just a guessing machine without understanding. The prediction is indeed the goal of training. But to hit it accurately, the model must represent grammar, facts, and connections within its numbers. This also explains the limitation: fabricated facts arise because an incorrect word piece can very well sound probable.