Linear Probe

A computer program that has learned from examples stores its knowledge in huge lists of numbers. These numbers arise inside the program while it is processing a text or an image. From the outside, nothing about them is apparent: they are simply thousands of decimal numbers. A linear probe is a tool for finding out what is contained in these numbers. One takes the numbers from a particular point in the program and trains a particularly simple computational rule on them that is meant to answer a single question. For example: Is the sentence the program is currently reading in German or in English? If this works reliably, the answer was already clearly contained in the numbers.

What the test reveals about the inner workings

Large AI systems are hard to see through. One sees the input and the output, but not why the system decided the way it did. This is precisely why there is an entire branch of research that examines models from the inside. The linear probe is one of the oldest and most widely used tools there, because it is fast and cheap.

Its appeal lies in its simplicity. The probe is only allowed to weight and add, nothing else. It therefore cannot work out anything complicated on its own. If it nevertheless delivers good results, this is not due to its own cleverness, but because the information is stored in the model cleanly and is easily accessible. Experts say the property is “linearly represented”.

This also allows practical questions to be clarified. For instance, researchers test at which processing stage a language model has grasped the grammar of a sentence. Or whether a model internally keeps track of whether a statement is true or false. Such findings later help to detect misbehavior early on.

From inside the model to a prediction

The procedure is always the same. First, many examples are sent through the finished model, say thousands of sentences. For each example, the numbers arising at a fixed point in the model are captured. This list of numbers is called an activation. For each example, the correct answer to the question being investigated is also noted down.

Then the probe is trained. It multiplies each of the numbers by its own weight, adds everything up, and decides based on the result. One can imagine this like calculating a grade: each subject counts to a different degree, and in the end there is a single number. During training, these weights are adjusted until the probe is right as often as possible. Important: the large model itself remains completely unchanged in the process.

Afterwards, the hit rate is measured on new examples that the probe has never seen. A well-known misconception lurks here. A high hit rate does not prove that the model actually uses this information. It only proves that the information is present. To clarify the difference, researchers additionally intervene in the numbers and observe whether the model’s output changes.

Probes in research and products

Linear probes regularly appear in technical articles on AI safety. Providers such as Anthropic or OpenAI report that they use them to search for internal signals that point to risky answers. Such a probe can run alongside the model during operation, since it costs almost no computing time.

The technique is also widespread outside of safety research. Anyone who wants to use a large image model for their own task often simply attaches such a simple layer and trains only that. This is considerably cheaper than retraining the entire model. In this context, one speaks of “linear probing” as an alternative to full retraining, i.e. fine-tuning.

The linear probe should be distinguished from explanation methods that highlight individual input words. These ask which part of the input was important. The probe instead asks what knowledge resides at a particular point in the model. Both perspectives complement each other, but answer different questions.