Interpretability

Modern computer programs that learn from examples instead of following fixed rules often deliver remarkably good results. The only problem is that no one can simply read off how they go about it. They consist of millions or billions of numbers that were automatically set during training. Together, these numbers produce behavior that no person ever wrote down. Interpretability is the technical term for the question of whether this behavior can nonetheless be understood and explained. It is sometimes also called explainability or explainable AI.

When an answer is correct but for the wrong reason

A system can be right and still be completely off track. A well-known example: an image recognition system was supposed to distinguish wolves from huskies and did so almost flawlessly. It later turned out that it was not paying attention to the animals at all. It had learned that pictures of wolves usually have snow in them. Without a look inside, this error would never have been noticed.

Such shortcuts become dangerous as soon as real decisions depend on them. Banks use learning systems for credit decisions, clinics for evaluating X-ray images, authorities for pre-sorting applications. Anyone who is rejected has a legitimate interest in an explanation. The European AI Act therefore requires that decisions in such high-risk applications must be traceable.

On top of this comes a safety aspect. The more tasks are handed over to language models, the more important the question becomes of whether they are really pursuing what they were instructed to do. A model could appear well-behaved during testing and behave differently in deployment. This is hardly distinguishable from behavior alone. Only a look at the internal processes could reveal the difference.

Tools for looking inside a model

The simplest approach measures which parts of the input influence the result the most. Individual words or image regions are experimentally altered and the resulting shift in the answer is observed. This produces a kind of heat map of the important spots. It shows what the model paid attention to, but not why that spot mattered to it.

A second approach deliberately builds simple models. A decision tree presents a chain of yes-no questions and can be read from top to bottom. Such methods are inherently interpretable, but noticeably weaker on complex tasks. This is the classic trade-off: more performance usually means less transparency. This relationship, however, is not entirely fixed.

The most ambitious approach is mechanistic interpretability. It attempts to decipher individual computational steps inside a language model, much as biologists study individual nerve cells. Research teams have in this way found building blocks that respond to specific concepts, such as the Golden Gate Bridge or programming errors. If such a building block is artificially amplified, the model’s behavior changes visibly. This is impressive, but so far covers only a tiny fraction of a large model.

From chatbot justifications to regulatory authorities

In everyday life you often encounter this topic without the technical term. When a chatbot backs up its answer with source links, that is a piece of traceability. When a streaming app explains why it is suggesting a particular movie to you, that too counts. One distinction is important here: when a language model explains its own reasoning in words, that is not genuine interpretability. This explanation is generated in the same way as any other text and does not necessarily correspond to what actually happened internally.

In the news, interpretability tends to come up in two main contexts. One is regulation, for instance when regulatory authorities require audits of credit-scoring or application software. The other is AI safety research, where labs such as Anthropic, OpenAI, and Google DeepMind maintain dedicated teams for this purpose. For companies, this has also become a cost factor by now, since documentation and auditability require effort.