Tokenization

Tokenization

Tokenization is the step in which a text is broken down into small units — so-called tokens — before being processed by an AI language model. These units can be whole words, syllables, or individual characters, and they are the actual basis with which a language model computes.

Before an AI language model can understand a text, that text must be broken down into manageable pieces. These pieces are called tokens. A token is usually a short word, a syllable, or a common word fragment — for example, “playing” is often split into “play” and “ing.” The model never sees letters or entire sentences at once, but always these tokens. Tokenization is thus the preliminary step without which no language model can work.

Why tokens and not simply words

The obvious idea would be to simply cut every text into words. In practice, this works poorly. Languages are irregular: capitalization, hyphens, typos, and foreign words create thousands of variants of the same term. A model that needed a separate entry for every variant would become huge and slow.

Tokenization solves this with a compromise. Common words like “and” or “the” get their own token. Rare or long words are broken down into smaller, common components. This keeps the total number of tokens manageable — typical models work with a vocabulary of 30,000 to 100,000 tokens. That is large enough for almost any language, but small enough to compute efficiently.

A widespread method is called Byte Pair Encoding, or BPE for short. It analyzes a huge body of training text and automatically searches for character sequences that occur together especially frequently. These are then turned into their own token. No one manually decides which syllables or fragments make sense — that emerges from the frequency in the text.

How a text becomes tokens

The process happens in milliseconds before the actual model responds. First, the raw text is normalized: spaces, punctuation marks, and special characters are treated consistently. Then the tokenizer — a special program — searches from left to right for the longest text fragment that appears in its vocabulary and replaces it with a number. In the end, there is no longer a character string, but a list of numbers.

An example: the sentence “I like eating chocolate ice cream” could be broken down into the tokens [“I”, “like”, “eating”, “choco”, “late”, “ice cream”]. The model then computes with the numbers corresponding to these tokens — for instance [482, 1091, 7823, 3041, 209, 55]. The order is preserved, but the content is now accessible to a computing machine.

Important: every model has its own tokenizer. The same text results in different token lists for GPT-4 and another model. This has consequences: a model trained on English performs worse on German text — because its tokens are optimized for English word structure and German words fall apart into many small fragments.

Tokens in everyday use — prices, limits, and language differences

Anyone accessing a language model via an interface, i.e., an API, usually pays per token — both for the incoming text and for the response. Providers like OpenAI publish precise price tables for this. A rule of thumb: 1,000 tokens correspond to roughly 750 English words. In German it is often fewer, because long compound words like “Rinderkennzeichnungsüberwachungsaufgabenübertragungsgesetz” consume many tokens.

Every language model also has what is called a context window — an upper limit on how many tokens it can process at once. Early versions of GPT-3 managed 4,096 tokens, newer models process over a million. The longer a document, the more tokens are needed. Anyone wanting to summarize a long PDF must check whether it fits within the context window — otherwise the model cuts it off or processes the text in parts.

In product reviews, tokenization usually appears indirectly: in figures on context lengths, prices per million tokens, or in benchmarks measuring how well a model handles a particular language. Anyone who understands what a token is can make sense of such figures — and recognizes why not every model works equally well in German.

Related Products

Latest News

Subscribe free. Unsubscribe the second it sucks.

High-signal news across AI, business, UX, and tech. Every morning.