Interpretability

Modern AI programs don’t learn from rules that humans wrote down. During learning, they adjust billions of numbers themselves until they get the right answer across many examples. These numbers can be read out, but no one understands them simply by looking at them. The system thus delivers an answer without it being clear why exactly this answer. Interpretability is the branch of research that wants to change exactly that: it searches for ways to describe the internal workings of such a system. The goal is an explanation that a human can check and, if in doubt, refute.

When No One Can Say Why: No

An opaque system is a problem wherever decisions have consequences. A bank uses a model to evaluate credit applications. The application is rejected. The customer asks for the reason, and the bank cannot name it. In the EU, this is not just uncomfortable but legally precarious. The AI Act requires that for systems in sensitive areas, decisions be documented in a traceable way.

The second reason is safety. A model can pick up patterns from the training data that no one intended. A well-known example from image recognition: a program was supposed to distinguish wolves from huskies and seemed very accurate. An analysis showed that it was actually paying attention to snow in the background. On the test dataset, this worked great. In reality, it would have collapsed immediately.

There is also a third point, one that particularly concerns very large language models. A model can appear unremarkable in tests and yet react differently in a real-world case. Anyone who only checks the outputs won’t see this. Anyone who can look inside the system has a chance to detect such tendencies beforehand.

From Feature Weights to a Look Into the Circuits

The simplest form is the question: which input influenced the result the most? Methods such as SHAP or LIME answer this by altering individual inputs and observing how the answer shifts. With images, this produces a heat map showing which image regions were decisive. This is useful but remains a description from the outside. It says what the model reacts to, not how it computes internally.

The more elaborate direction is called mechanistic interpretability. Here, researchers try to understand individual building blocks inside the model like components of a circuit. A well-known result are so-called induction heads: building blocks that recognize when a pattern repeats in the text and continue it. They can be deliberately switched off and one can observe which capability is lost as a result. That is the difference between a conjecture and a proof.

A central obstacle here is so-called superposition. A single building block usually does not stand for a single concept but participates in many at once. There are simply more concepts than there are building blocks. Newer methods therefore try to computationally decompose these mixed signals back into individual, nameable features. In this way, Anthropic has made millions of such features visible in a language model, including features for programming code errors or for specific cities.

Where Explainability Is Demanded

In medicine, interpretability has long been a prerequisite for deployment. A system that flags abnormalities on X-ray images must show which part of the image it means. Only then can a doctor even assess the hint. It’s similar in lending, insurance, and application processes.

In tech news, the term is mostly encountered in connection with AI safety. When labs like Anthropic, OpenAI, or Google DeepMind report on progress, it is often about detecting undesirable behavior on the inside. A common misconception here is that a chatbot can explain itself. If you ask it for its reasoning, it generates a plausible-sounding answer. Whether this has anything to do with the actual internal computation is not thereby established.

One should also not overestimate the claim. Complete interpretability of large models does not exist so far. What does exist are partial explanations for individual capabilities and tools for specifically searching for known problems. That is significantly more than five years ago, but far from a blueprint one could simply read.