Q4_K_M

At its core, an AI language model consists of billions of stored numbers. These numbers determine how the model calculates a response from an input. Originally, each of these numbers is stored quite precisely, at 16 bits per number. A bit is the smallest unit of storage, that is, a single zero or one. Q4_K_M is a method for storing these same numbers more coarsely: on average with about 4.5 bits instead of 16. As a result, a model file shrinks to roughly a quarter of its size, without the responses noticeably getting worse.

Why 40 gigabytes suddenly become 12

Large language models are above all a storage problem. A model with 20 billion numbers takes up around 40 gigabytes at full precision. Hardly any normal PC has that much graphics memory. In the Q4_K_M version it’s about twelve gigabytes, which means the same model fits on a decent gaming graphics card or a newer laptop.

This changes who can even use such models. Without shrinking, you need a data center or a subscription with a provider. With shrinking, the model runs locally, that is, on your own device. The texts you enter then never leave the computer, which is a strong argument for companies with sensitive data.

One side effect is speed. The computer has to read all the model’s numbers from memory for every response. Less data means less reading work, which is why a shrunken model often responds noticeably faster. The price is a small loss of quality, which for Q4_K_M, according to the usual measurements, lies in the range of a few percent.

What the three parts of the abbreviation mean

The name is not a brand but a description. Q stands for quantization, that is, converting precise numbers into coarser levels. You can picture this like a thermometer that, instead of showing 21.37 degrees, only shows whole degrees. The 4 indicates that most numbers are stored with four bits, meaning they only have 16 different levels.

The K stands for an improved method called K-quants. Here, the numbers are split into small blocks. For each block, an additional scale is stored that determines how coarse the 16 levels turn out to be within that block. This way, blocks with very small values still remain usably precise.

The M at the end stands for medium and describes the mix. Not all parts of a model react equally sensitively to imprecision. Q4_K_M stores particularly sensitive parts with six bits instead of four. There are also the variants S for small and L for large, which contain fewer or more of these more precise parts. Q4_K_M is considered the compromise that best balances size and quality.

Q4_K_M in download lists and model names

Most often, you encounter the abbreviation as part of a file name. On platforms like Hugging Face, a kind of app store for AI models, a dozen files are often available for a single model. They’re then named something like modelname-Q4_K_M.gguf. GGUF is the file format in which shrunken models are stored for home use.

Programs like Ollama or LM Studio, which let you run models on your own computer, often preselect Q4_K_M automatically. In forums and reviews, this variant therefore serves as the standard comparison: when someone writes that a model runs at 30 words per second, this is usually the version they mean.

A common misconception is that a shrunken model is dumber than a model that was small to begin with. The opposite is usually true: a large model in Q4_K_M beats a small model at full precision almost every time, at a similar file size. Given the choice, it’s therefore better to pick the larger model in coarser storage.