Discrete Tokens

Computers cannot work directly with language, images, or music. For this, they need numbers. One approach is to cut the content into individual building blocks that come from a fixed, predetermined list. Exactly such building blocks are called discrete tokens. “Discrete” here means: countable and clearly separated, with no intermediate stages. A text then consists of, say, building block number 4711, followed by number 92 and number 305 – each piece is either fully present or not present at all.

Why language models think in building blocks

A modern language model like ChatGPT predicts, at every step, which building block comes next. This only works because the selection is finite. Typical models operate with a list of roughly 50,000 to 200,000 possible tokens. The model assigns a probability to each entry in this list and chooses from among them. If there were infinitely many possibilities, this probability distribution could not be computed at all.

The opposite pole is continuous values. The brightness of a pixel or the amplitude of a sound wave can be graded arbitrarily finely. Such quantities cannot be numbered off one by one. That is why researchers often first convert images and sounds into discrete tokens before a language model processes them. This allows text, image, and audio to be handled with the same technique.

For users, there is a very practical reason to look into this: providers bill their prices per token. The maximum length of a conversation, the so-called context window, is also measured in tokens. Anyone who knows what a token is understands their bill and the technical limits of a model.

From sentence to sequence of numbers

The breakdown is handled by a program called the tokenizer. It uses a fixed dictionary that was built from huge amounts of text before training. Frequent words get their own entry in it. Rare words are broken down into several parts. “House” is usually one token, whereas “Danube steamship company” might be five. Spaces and punctuation marks are counted too.

As a rule of thumb in German: one token corresponds to roughly three to four characters. German texts tend to need somewhat more tokens than English ones, because the dictionaries were mostly built from English material. Long compound words add to this cost.

Every entry in the dictionary has a number. However, this number alone says nothing about meaning. That is why, for every number, the model looks up a long vector of numbers, the so-called embedding. Only this vector carries the meaning. You can imagine the dictionary as a printer’s case with lead letters: there is a fixed number of compartments from which any text at all is typeset.

Visible consequences in everyday chatbot use

A well-known example: language models sometimes fail at the question of how many “r”s are in the word “Strawberry.” The reason lies in the breakdown. The model does not see individual letters, but two or three word fragments. Counting letters is then like being asked to count the fibers in a sheet of paper without putting it under a magnifying glass.

In the news, tokens mainly show up in pricing tables and performance specs. Reports like “one million tokens of context” or “$0.50 per million tokens” always refer to these building blocks. One million tokens corresponds roughly to a stack of several thousand book pages.

The principle has also become common outside of text. Image generators and music models break their data down into discrete tokens, often using a method called vector quantization. Incidentally, a common misconception is to equate tokens with words. The boundaries run differently, and even a single additional space can change the breakdown.