GGUF

A program that understands language and writes texts consists at its core of many billions of numbers. These numbers are the result of a long learning process and have to be stored on disk somehow. GGUF is a fixed rule for exactly how these numbers are arranged within a file. Besides the numbers themselves, such a file also contains extra information: the name of the model, its size, and hints on how a conversation with it needs to be structured. Everything is packed into a single file with the extension .gguf, which can be downloaded just like a photo or a video. The name is an abbreviation and roughly stands for a unified format of the GGML software library.

Why models fit on your own computer because of it

The best-known AI services run on expensive servers owned by a company. Every input is sent there and processed there. GGUF is the main reason why things can also work differently. Anyone who downloads such a file can start the model on their own laptop, without an internet connection and without an account.

This has concrete consequences. The texts entered never leave the computer, which can be crucial for doctors, lawyers, or companies with trade secrets. There are no ongoing costs per request. And nobody can shut down or alter the model afterward, because the file sits on your own hard drive.

A second point is uniformity. Before GGUF, there were many competing formats, and a file often only worked with exactly one program. Today, almost all tools for local AI accept the same format. You can download a file once and use it with different programs.

Coarser numbers, smaller files

The crucial trick is called quantization. Originally, each of the billions of numbers is stored very precisely, usually using 16 bits of storage per number. During quantization, they are rounded more coarsely, often down to just 4 bits. As a result, the file shrinks to about a quarter of its size. A model that previously took up 16 gigabytes then fits into roughly 4 gigabytes, and thus into the memory of an ordinary laptop.

You can think of this like an MP3 file. There, too, information is discarded to save space, and the result still sounds usable. Likewise, a heavily rounded model loses some quality. It makes more computational errors and sometimes phrases things less precisely. With moderate rounding, though, the difference is barely noticeable in everyday use.

That’s why the same model is usually available in several GGUF variants with labels like Q4 or Q8. The number roughly indicates how many bits per number remain. Anyone with a lot of memory takes a large variant; anyone with little memory takes a small one. Another advantage of the format: the program doesn’t have to read in the whole file but can selectively load the parts it currently needs. This makes startup faster.

Local AI programs and model downloads

GGUF is most commonly encountered in programs used to run AI models yourself. Well-known examples are llama.cpp, Ollama, and LM Studio. You pick a model there, the program downloads the matching GGUF file, and afterward you can type into a chat window as usual. On the platform Hugging Face, a kind of public warehouse for AI models, tens of thousands of such files are available.

In the news, the term usually comes up when a company releases a model openly. It is then often said that GGUF versions appeared within hours. This means: volunteers converted and quantized the model so that anyone can try it out at home.

A common misconception is that GGUF itself is an artificial intelligence or a program. It is merely the packaging. Without a program that opens the file and carries out the computations, nothing happens with it at all. And a second note: a GGUF file can generate text, but it doesn’t learn anything new. Training is completed before the file even comes into existence.