Counter-Metric

Companies and software teams usually measure their success by a single primary number. For a video app, that might be the time people spend watching videos. But such numbers can often be increased in unpleasant ways. You can raise watch time by showing increasingly sensational content until users feel bad afterward and delete the app. A counter-metric is the second number that specifically guards against this: it measures the harm that could arise from chasing the primary number. The primary number is supposed to rise, while the counter-metric must not get worse in the process.

Why a single success number is dangerous

As soon as a number becomes a target, it changes the behavior of the people being measured by it. A support team that is evaluated by the number of tickets handled closes cases quickly and sloppily. A recommendation system optimized for clicks learns to produce sensationalist headlines. In both cases the number goes up while the actual goal is missed.

This effect is known as Goodhart’s Law: a metric loses its informative value as soon as it becomes the target. Counter-metrics are the practical way of dealing with this. You don’t pretend to have found the one perfect number. Instead, you pit two numbers against each other that cannot both be cheaply improved at the same time.

This is especially important in AI systems, because there a program optimizes on its own. A human eventually notices when they’re doing something nonsensical. A model trained toward a target number does not notice. It reliably finds exactly the shortcuts nobody wanted.

How to choose a sensible pair of numbers

A good counter-metric measures precisely the side effect that the simplest abuse of the primary number would produce. So you first ask yourself: how could I dishonestly increase this number? Whatever would break in the process is what gets measured. For watch time, a fitting counter-metric is the number of users who still open the app after a week. For the number of support tickets handled, a fitting counter-metric is the share of customers who contact support again about the same issue.

Both numbers are observed side by side, for example in an A/B test. Here, one half of users gets the new version, the other the old one. Beforehand, the team sets a threshold: the primary number must rise by at least a certain amount, and the counter-metric may fall by no more than a small amount. If the threshold is violated, the change is considered a failure, even if the primary number looks great.

A common mistake: you can’t just take twenty counter-metrics and hope that nothing gets overlooked that way. With very many numbers, some of them will always look bad, purely by chance. Then the team ends up debating endlessly instead of deciding. Two to four well-chosen counter-numbers are the common practice.

Counter-metrics in products and quarterly reports

In the development of large language models, this pattern is firmly built in. A model is supposed to respond more helpfully, so the primary number is user satisfaction. As a counter-number, teams measure how often the model delivers fabricated facts or complies with dangerous requests. Such safety numbers are also called guardrail metrics, because they act like a guardrail.

You also encounter this principle in financial news, often without the English term. When a company reports rising revenue, analysts immediately look at the profit margin and customer churn. Revenue can be bought with discounts, so on its own it’s worth little. The second number determines whether the growth is healthy.

For you as a reader, this is a useful check to apply. Whenever only a single success number is presented somewhere, the counter-number is usually missing. The question then is: which number would I need to see to recognize that this success was bought at a high price?