Tokenizer

A computer can’t do anything with letters. It only computes with numbers. So for a program that writes or understands text to even get started, language must first be translated into numbers. That’s exactly what the tokenizer does: it cuts the text into small pieces and gives each piece a fixed number from a long list. These pieces are called tokens. A token is sometimes a whole word, but often just part of a word or a punctuation mark. The sentence “The dog sleeps.” thus becomes maybe five or six numbers, and the program computes with those from then on.

Why tokens determine the bill

Tokens are the unit of measurement in the AI industry. Anyone who integrates a service like ChatGPT into their own software via an interface pays per token. Both the input text and the response are billed. A price of “1 dollar per million tokens” roughly means: one million tokens is about 750,000 English words. How a tokenizer cuts text therefore directly determines cost.

A model’s memory is also measured in tokens. The so-called context window indicates how much text a model can take in at once. If it says “128,000 tokens,” that’s roughly enough for a thick book. Anything beyond that gets dropped or has to be shortened.

And tokenizers aren’t equally fair to all languages. Most large models were tuned mainly on English text. English words therefore often need just one token, while German compound words like “Rechtsschutzversicherung” need five or more. The same content in German thus costs more money and more space in the context window. For languages like Thai or Amharic, the difference is even greater.

How the text gets cut up

Naively, one could simply split on spaces, giving one token per word. That fails quickly. The list of all possible words would be huge, and with typos or new words like “climate glue-on” the model would be at a loss. The other extreme, one token per letter, always works, but makes texts extremely long and expensive.

That’s why modern systems use a middle path, usually a method called byte pair encoding. Here, the tokenizer is pre-trained on large amounts of text. It starts with individual characters and repeatedly merges the most frequent neighboring pair into a new building block. Common words like “and” thus end up as a single token in the list. Rare words remain split into parts, for example “climate” plus “glue-on”.

The result is a fixed vocabulary, typically between 30,000 and 200,000 entries. Importantly, it’s a fixed part of the model. Anyone who swaps out the tokenizer afterward renders the model useless, because the same numbers would then suddenly mean different pieces of text.

What models are therefore surprisingly bad at

Many well-known weaknesses of chatbots trace back to the tokenizer. Ask how many “r”s are in “strawberry” and models often get it wrong. The reason is simple: the model doesn’t see letters, but two or three numbered word chunks. It would have to guess what’s inside its own building blocks. Rhymes, syllable counting, and arithmetic with long numbers are similarly unreliable.

In everyday use, the term mostly appears in price lists and spec sheets. Providers like OpenAI, Google, or Anthropic state prices and context lengths exclusively in tokens. There are websites, so-called tokenizer demos, where you type in a sentence and see the split marked in color. That’s the fastest way to get a feel for the building blocks.

The topic also comes up in trade news. When a company announces that its new model processes text “30 percent more efficiently,” sometimes it’s really just a better tokenizer behind it. And research into tokenizer-free models that work directly with individual bytes is considered an exciting alternative. So far, however, the classic tokenizer has remained the standard.