Language Token

Computers can’t do anything with letters. They only compute with numbers. So, for a text program to process a sentence, the sentence must first be cut into pieces and then translated into numbers. These pieces are exactly what are called language tokens. A token is usually a short word, a word fragment, or a punctuation mark. The sentence “Today is Monday.” consists, for example, of four tokens: “Today”, “ is”, “ Monday”, and “.” – the space often belongs to the token as well.

Why text breaks down into chunks and not into words

One might get the idea of simply treating each word as a unit. This fails because of the sheer quantity. In German alone there are millions of possible word forms, plus names, technical terms, and newly coined words. A program would have to store a separate entry for every single form. New or misspelled words would simply be invisible to the system.

The other extreme case would be to take each letter individually. Then you’d get by with just a few dozen characters. But then a normal sentence would consist of a great many units, and the program would have to work out anew what each word meant every single time. Both approaches cost computing time. Tokens are the compromise between these extremes: common words get their own token, rare ones are broken into pieces.

This breakdown also explains a well-known weakness of chatbots. If you ask a language model how many letters “r” are in the word “strawberry”, it surprisingly often gets it wrong. The reason is mundane: the model doesn’t see individual letters, only two or three word building blocks. The letters within them are barely visible to it.

How a tokenizer cuts up a sentence

Responsible for the breakdown is a separate program, the tokenizer. It has a fixed list of allowed text pieces, usually between 30,000 and 200,000 entries long. This list is called the vocabulary. The tokenizer goes through the text and always looks for the longest piece that is on its list. It then replaces each piece with its number on the list.

The vocabulary is not written by hand but generated from large amounts of text. One method starts with individual letters and then repeatedly merges the most frequent pair of letters into a new unit. After many rounds, “e” and “n” have thus become the ending “en”, and frequent syllables have become whole words. What occurs often becomes a token; what is rare stays broken up.

This has an unfair side effect. The vocabularies of the large models were predominantly built from English texts. English words therefore often need only one token, German words two or three. For long German compound words like “Donaudampfschifffahrt” it can easily become six or more. The same content thus costs more tokens in German than in English.

Tokens on the bill and in the context window

Anyone using an AI via a programming interface pays by the token. Providers state their prices per one million tokens, separated by input and output. Output tokens are regularly considerably more expensive, because the model has to generate them one after another individually. As a rough rule of thumb: one standard page of text corresponds to roughly 500 to 700 tokens.

The second major metric of language models is also measured in tokens. The context window indicates how much text a model can keep in view at once. Modern systems advertise 128,000 or even one million tokens. That corresponds to several hundred pages of a book. Once the window is full, the beginning of the conversation falls out the back – which is why chatbots sometimes forget in long conversations what was discussed at the start.

The term therefore constantly appears in tech news, usually as a price or performance figure. If the price per million tokens drops, running AI applications becomes cheaper. If the context window grows, longer documents can be processed at once. Incidentally, the language token is not to be confused with the token from the world of finance, which denotes a digital unit of value. The two terms have nothing to do with each other.