Model Compression

At its core, an AI model is a huge collection of numbers. These numbers determine how the program computes an answer from an input. In large language models, there are hundreds of billions of such numbers, and each one takes up storage space. A single model can thus be several hundred gigabytes in size, more than the hard drive of many laptops. Model compression is the umbrella term for all techniques that shrink or simplify this quantity of numbers. Afterwards, the model should be able to do the same things as before, just with less effort.

What costs a smaller model saves

Large models normally run on specialized chips in data centers. These chips are expensive and scarce, and they consume a lot of electricity. Every single answer a chatbot gives therefore costs the operator money. If a model needs only half as much memory, it fits on cheaper hardware. With millions of requests per day, this adds up to substantial amounts.

The second reason is the location of execution. A compressed model can run directly on a smartphone or in a car. Then a request no longer has to be sent over the internet to a remote server. That is faster, works without a network connection, and is better for privacy, because private data never leaves the device.

It is important to distinguish this from a related idea: compression does not mean training a small model from scratch. One always starts from an already finished, large model and shrinks it afterwards. That is considerably cheaper than letting a new training run go on for weeks.

The four common shrinking techniques

The most common technique is called quantization. In this process, the numbers in the model are stored more coarsely, for example with 8 instead of 16 digits of precision. You can think of it like a weather forecast: whether 21.4 degrees or 21 degrees is announced changes little about what you decide to wear. The same holds true for most computational steps in the model – the coarser number is sufficient.

In pruning, entire connections within the model are removed. Many numbers lie close to zero and contribute barely anything to the result. They are removed, and afterwards the model is briefly retrained so it gets used to the gaps. Related to this is factorization, in which large number tables are replaced by several smaller ones that together achieve almost the same thing.

The fourth technique is knowledge distillation. Here, a large model trains a small one by having the small one try to imitate the answers of the large one. The technical term for this is the teacher-student principle. All four techniques can be combined. In practice, each stage costs some accuracy, and developers look for the point at which the loss is still just about acceptable.

Compressed models in products and headlines

When a provider releases a model in several sizes, say with 7, 13, and 70 billion numbers, distillation is often behind it. On platforms like Hugging Face, a kind of app store for AI models, you can additionally find quantized versions. Name suffixes like “4-bit” or “INT8” reveal how heavily a model was compressed. Such versions run on an ordinary gaming PC.

In everyday life, you encounter the result without anyone using the term. Speech recognition without an internet connection, offline translation, facial recognition when unlocking your phone – compressed models are behind all of these. Assistance systems in cars also work this way, because a delay caused by the wireless connection would not be acceptable there.

In business news, the topic usually comes up in connection with costs and chip demand. A common misconception is that a compressed model is simply a stripped-down, cheap version. With good implementation, the difference in quality on normal tasks is barely measurable. It only becomes noticeable on difficult tasks, such as long chains of reasoning or rare languages.