Contrastive Pruning

Programs that understand language or recognize images consist of millions of small adjustment screws. Many of them contribute barely anything to the result. Deleting them is called pruning — like cutting the dead branches off a tree. Contrastive pruning is a particular way of making this decision. Instead of showing the program only individual examples, one always shows it pairs: two very similar inputs that nevertheless must be treated differently. Only the adjustment screws that are needed for this difference are allowed to remain.

Why small models must not become stupid

Large AI models are expensive to operate. Every request costs computing time, and therefore electricity and money. A model that shrinks to a third of its size runs correspondingly cheaper. It also fits on devices with little memory — a phone, for example, or a control unit in a car.

The catch with ordinary pruning: one usually only measures the average hit rate. A pruned model can continue to be correct in 95 percent of all cases and still lose precisely the difficult cases. These are the cases in which two inputs look almost identical but require different answers. “The film was not bad” and “The film was bad” differ by one word. A model that has been pruned too coarsely overlooks exactly this word.

Contrastive pruning addresses this. It evaluates components not by whether they are useful on average, but by whether they keep similar cases apart. This preserves fine distinctions even though the model shrinks considerably. This is especially important in areas where mistakes are costly: with medical images, for instance, or with fraud detection in payment transactions.

Contrasting pairs as a touchstone

The process begins with a fully trained model. One builds pairs of inputs: a positive pair that has the same meaning, and a negative pair that differs only minimally but must be classified differently. Two photos of the same cat from a different angle would be a positive pair. A photo of a cat and one of a very similar-looking lynx would be a negative pair.

Now one looks at which internal components react differently to these pairs. A component that delivers exactly the same signal for both cat and lynx does not help with the distinction. It is a candidate for deletion. A component whose signal diverges markedly between the two images carries the distinction — it stays. This produces a ranking of importance.

After the deletion, a brief retraining almost always follows, known as finetuning. The model is allowed to readjust itself using its remaining components. A common misconception is that pruning is the same as quantization. With quantization, all components are retained; only their numbers are stored more coarsely. With pruning, components disappear entirely. In practice, the two are combined.

Where shrunken models turn up

The procedure itself is rarely visible, but its results are constantly present. When a provider releases a smaller version of its language model, it is almost always based on a mix of pruning and related shrinking techniques. The speech recognition that runs on a phone without an internet connection is one such case. Translation features in browsers likewise work with heavily reduced models.

In trade articles and company announcements, the term usually appears in the context of “model compression” or “efficiency.” Anyone reading such reports should pay attention to two figures: by what percentage the model has shrunk and how much accuracy was lost in the process. Figures such as “50 percent smaller with less than one percent accuracy loss” are typical statements.

For investors and observers of the industry, the topic is relevant because it directly affects the operating costs of AI services. A company that can halve its models roughly halves its computing bill as well. Contrastive pruning is not a miracle cure in this regard, but one of several levers — albeit one that specifically protects the difficult cases.