Tokenizer Fingerprinting

A language model cannot do anything with letters alone. Before processing, it splits every text into small building blocks called tokens. A token is usually a word fragment, for example “bi” and “cycle” instead of “bicycle”. Which building blocks exist is fixed once and does not change afterward. This list differs somewhat from one provider to the next. Tokenizer fingerprinting exploits exactly that: you look at the splitting and use it to identify which model or model family is behind it.

What the fingerprint reveals about a model’s origin

Many AI models today are released as ready-made files. Others can only be reached via an interface on the network, without anyone being allowed to look inside. In both cases, it is often unclear what a model is really built on. Companies like to claim they developed it from scratch. The fingerprint of the building-block list can support or undermine that claim.

The reason is simple: changing the building-block list is extremely costly. Anyone who takes over an existing model and merely fine-tunes it almost always keeps the list unchanged. It thus survives modifications that would otherwise erase many traces. This makes it one of the most stable identifying features there is.

This has concrete consequences. In licensing disputes, the question is whether a provider is passing off someone else’s work as their own. In security reviews, one wants to know whether a purchased AI is secretly based on a model from another country. Journalists and researchers also reach for this tool when a start-up makes big promises.

How the splitting is made visible

It is easiest with openly available models. There, the building-block list comes as a file and can be compared directly. Two models with an identical list very likely share a common origin. Even the order of the entries is telling, since it arises during the construction of the list and is practically never coincidentally the same.

It gets harder when the model can only be addressed through an interface. Then you work with test inputs. You send rare character sequences, such as emojis, Chinese characters, or made-up words. You then observe how many tokens are billed or how the model reacts. From these reactions, the splitting can be reconstructed piece by piece.

A related but different approach is the watermark. There, a provider deliberately builds in an identifying feature. With fingerprinting, on the other hand, you use traces that arise anyway. No one needs to prepare anything in advance. A common mistake is to treat the fingerprint as proof. It is a strong indication, nothing more, since two teams can independently use the same freely available building-block list.

Where the topic turns up in the news

Whenever a new model turns out to be surprisingly good, the same scrutiny begins in expert forums. Within a few hours, developers compare the building-block lists with known models. Several times it turned out that supposed in-house developments were actually based on open models from Meta, Alibaba, or Mistral. Such findings then end up in the tech press and cost companies trust and sometimes customers.

The splitting also plays a role in pricing questions. Providers bill per token, not per word. German texts with long compound words break down into more building blocks than English ones. The same text therefore costs different amounts with different providers. Anyone comparing such costs is already unknowingly working with the fingerprint.

For companies, the method is increasingly becoming part of procurement due diligence. Before buying an AI solution, one wants proof of what is inside it. Regulation such as the European AI Act requires information about a model’s origin. Tokenizer fingerprinting is one of the few methods that can be used to verify such information from the outside.