Jacobian Lens

An AI system takes an input and produces an output. In an image program, the input is a photo, and the output might be the word “cat.” From the outside, one only sees these two ends, not the path in between. The Jacobian Lens is a tool researchers use to indirectly measure this path. It alters the input by a tiny amount and observes how much this shifts the output. Many such measurements together produce a kind of sensitivity map of the system.

What the sensitivity map reveals about a model

Modern AI models consist of billions of numerical values. No one can read these values individually and understand from them why the model answers one way and not another. This is often called the black-box problem: you see the result, but not the reasoning. Methods like the Jacobian Lens are an attempt to at least partially open this box.

The practical benefit lies mainly in identifying errors. A medical imaging model reliably detects tumors — but perhaps it is actually reacting to the clinic logo in the corner of the image. A sensitivity measurement would uncover this, because the output changes drastically as soon as that corner of the image is touched. Such spurious correlations are one of the most common reasons AI systems perform well in the lab but poorly in everyday use.

A second reason is security. If a minimal change to the input completely flips the output, the system is vulnerable to attack. This is precisely the principle exploited by so-called adversarial attacks: they add noise invisible to humans and cause the model to produce a completely wrong result. Anyone who knows the most sensitive points can specifically secure them.

Small perturbations, measured response

The name comes from mathematics. The Jacobian matrix is a table that records, for each input quantity, how strongly it influences each output quantity. “Lens” here means a perspective through which one looks at the model. Together, then, a way of viewing the system through the lens of its sensitivities.

In practice, this works as follows: one takes an input, changes a tiny part of it, and lets the model compute again. The difference between the two outputs is the measured value. Repeating this for all parts of the input yields a complete picture. For an image, every pixel gets a value indicating how important it was for the decision. These values can be overlaid as a colored map on top of the original image.

A common mistake is to read the result as an explanation. The measurement only shows what the model reacts to, not why. It provides clues, not proof. Moreover, strictly speaking, it is only valid for very small changes around this one specific input. For a different photo, the map can look completely different.

From research paper to product

You are most likely to encounter this term in academic papers on AI interpretability — that is, on the question of how to make models understandable. It rarely appears verbatim in news reports. There, it’s more likely to say that a team “analyzed what a model pays attention to” — which often refers to exactly this kind of measurement.

In everyday life, one encounters the results indirectly. Some diagnostic software for doctors highlights in color which areas of an image contributed to an assessment. Regulatory and oversight bodies increasingly demand such evidence before an AI system may be deployed in sensitive areas. The European AI Act is moving in this direction as well.

It is important to distinguish this from related methods. The Jacobian Lens measures the finished model from the outside. The better-known Logit Lens, by contrast, looks into the intermediate layers of a language model and reads off which word is already being provisionally favored there. Both belong to the same toolbox, but they operate at different points.