SHAP

Many programs today make predictions that a human cannot follow step by step. A bank, for example, calculates how likely someone is to repay a loan. The result is a number, but no explanation. SHAP provides this explanation after the fact. It breaks down the result into the contributions of individual pieces of information, such as income, age, and payment history. In the end, there is a sentence like: The high income raised the assessment, while the short account history lowered it.

When a number alone is not enough

Predictive models are now used in situations where a great deal is at stake. This includes loans, insurance premiums, job applications, or medical findings. Those affected have a legitimate interest in learning why a decision turned out the way it did. A number without justification is worthless for that purpose.

Legal pressure is also growing. The European General Data Protection Regulation requires meaningful information about the underlying logic in the case of automated decisions. Following the EU’s AI legislation, systems used in sensitive areas must be documented in a traceable way. Methods such as SHAP are a common tool for practically meeting such requirements.

For developers themselves, SHAP is often even more useful. It reveals when a model gets the right answer for the wrong reasons. A well-known pattern: a system for detecting sick patients relies heavily on the hospital identifier. It has learned that a particular clinic has more severe cases, rather than what illness actually looks like. Such shortcuts only become apparent once you look inside the model.

The fair share of the outcome

The idea behind SHAP comes from game theory, a branch of mathematics. There is a classic question in that field: several people jointly earn a profit—how should it be divided fairly? The mathematician Lloyd Shapley found a solution to this in the 1950s. One considers all possible orders in which the participants join the group. For each order, one measures how much the newcomer contributes in addition, and averages these contributions.

SHAP applies this to predictions. The participants are not people but the individual input values, referred to in technical terms as features. The profit to be distributed is the gap between the average outcome across all cases and the outcome for this particular case. Each feature receives a value indicating how strongly it pulled the outcome up or down. These values sum up exactly to the difference — nothing is lost.

A complete calculation is only feasible with a small number of features, since the number of possible orders grows explosively. With twenty features, there would already be more than two quadrillion combinations. That’s why, in practice, SHAP works with approximations. For decision trees, there is a fast, exact method called TreeSHAP; for other models, estimates are made using sampling.

Where SHAP charts show up

SHAP is most commonly encountered in corporate data analysis. Banks, insurers, and online retailers use it to check their risk models and explain them to regulators. A typical visualization is a bar chart for each individual case: red bars pointing right for features that increase the outcome, blue bars pointing left for the opposite direction. Such graphics also appear in model reports and in coverage of AI regulation.

SHAP is freely available software and can be connected to common models with just a few lines of code. This explains its widespread use. In job postings for data analysis, the term regularly appears alongside tools like LIME, which pursues a similar goal via a different computational approach.

A common misconception is that SHAP shows causes. It does not. It shows what the model relies on, not what actually has an effect in reality. If a model has learned a relationship incorrectly, SHAP will explain this error just as cleanly and convincingly. With very large language models, the method also reaches its limits, since millions of input components interact there.