cl100k_base
cl100k_base is the dictionary that some OpenAI AI models use to break text down into small building blocks before processing it. It comprises around 100,000 such building blocks and was used by, among others, GPT-3.5 and GPT-4.
Computers calculate with numbers, not letters. A language program must therefore first translate written text into numbers. For this, there is a fixed list of text building blocks: every word, every syllable, and every punctuation mark on this list has its own number. cl100k_base is one such list. It contains just under 100,000 entries, hence the “100k” in the name. The company OpenAI created it, and the programs ChatGPT, GPT-3.5, and GPT-4 converted text into numbers using exactly this list for years.
Why the list of building blocks determines the price
Providers of AI services do not bill by words or characters, but by these building blocks. In technical jargon they are called tokens. Anyone who sends a request to a model pays for every token in the question and for every token in the answer. The list of building blocks therefore directly determines how expensive a text is.
However, cl100k_base is not equally economical for all languages. It was built predominantly on English text. English words therefore often fit into a single building block. German words break down into multiple parts more often, especially long compound words like “Krankenversicherung.” For languages with a different script, such as Thai or Hindi, it becomes even less favorable. The same content costs a multiple there.
A second reason is the context window. This is the amount of text a model can take in at once, measured in tokens. If a model processes 8,000 tokens, how many pages of text that actually amounts to depends on the list of building blocks. Efficient decomposition thus means more usable space.
How the 100,000 building blocks came about
The list was not written by hand but calculated from large amounts of text. The procedure is called Byte Pair Encoding, abbreviated BPE. It starts with individual characters and then looks for the pair of characters that appears next to each other most frequently. This pair is merged into a new building block. This step is repeated tens of thousands of times until the desired number is reached.
The result is a mixture of whole words and word parts. Frequent words like “and” or “ the” get their own entry. Rare words are composed of several pieces. A name the list does not know is, if necessary, represented letter by letter. This way, any text at all can be represented, including emojis or program code.
It is important to distinguish this from the model itself. cl100k_base learns nothing and understands nothing. It is pure lookup in a fixed table, comparable to a character set. The intelligence lies in the model behind it. That is why the same list can be used by several models, and a successor such as o200k_base with around 200,000 building blocks does not make a model smarter, only more economical.
Where the name turns up in tools and bills
Anyone who writes their own programs that call on AI models quickly comes across the name. OpenAI’s tiktoken software library expects you to specify the appropriate list. There, “cl100k_base” appears literally in the code. Many cost calculators online also count tokens according to this scheme.
In everyday use, the decomposition shows up in a well-known weakness. If you ask an older model how many “r”s are in the word “strawberry,” you often get a wrong number. The model does not see individual letters, only two or three building blocks. Spelling words backwards is also difficult for the same reason.
In technical articles, cl100k_base is nowadays mostly mentioned historically. Newer OpenAI models such as GPT-4o use the larger list o200k_base. Other providers have their own lists of building blocks anyway, such as Google for Gemini or Meta for Llama. That is why token counts can only be compared between providers to a limited extent. Anyone comparing prices should always check which decomposition lies behind them.