Expert Offloading

Many modern AI models are divided into layers and specialized subnetworks called experts. For any single computation, only a few of these are needed at the same time. Expert Offloading exploits exactly this: the experts that are not currently needed are removed from the graphics card's fast memory and cached elsewhere – for example in the computer’s regular main memory or even on disk. When they are needed, the system loads them back within milliseconds. From the outside, the model appears complete, even though never all of it resides in expensive graphics memory at once.

Why Expert Offloading makes the difference

Graphics cards have their own very fast memory – known as VRAM. It is expensive and limited. An upper-mid-range graphics card typically has 12 to 24 gigabytes of VRAM. A powerful AI model like Mixtral 8x7B, however, requires over 80 gigabytes to fit entirely in VRAM. Without Expert Offloading, this model simply cannot be run on ordinary home hardware.

With Expert Offloading, this boundary shifts considerably. Anyone with enough regular main memory – which is slower but much cheaper and often available in large quantities – can still run the same model. Responses take somewhat longer because loading the experts takes time. For many applications, this is an acceptable trade-off.

How the offloading process works in detail

The process follows a fixed pattern. First, a small control network – the so-called router – decides which experts are responsible for the current input. The system then checks whether these experts are already in VRAM. If not, they are loaded from main memory or disk. At the same time, experts that are not currently needed and are taking up space are offloaded.

What matters most is how well the system can predict which experts will be needed next. Modern implementations try to optimize the loading order and load experts proactively before they are actually needed. The better this prediction works, the less waiting time occurs. A common misconception is that Expert Offloading is simply the same as loading an entire model in parts – in fact, it involves a dynamic, request-dependent back-and-forth that is decided anew for each computation.

There is still a bottleneck, however: the transfer rate between storage locations. VRAM is many times faster than regular main memory, which in turn is significantly faster than disk. The farther away the offloaded experts are stored, the more noticeable the slowdown becomes.

Expert Offloading in practice

Expert Offloading appears mainly in projects aiming to make large language models runnable on ordinary home computers. The open-source project llama.cpp supports the technique, as do specialized runtime environments such as Ollama or llm.gguf. So when you read reports of someone running a huge model “on a cheap gaming GPU,” Expert Offloading is usually behind it – combined with other compression techniques.

The principle is also relevant in the enterprise world. Anyone who doesn’t want to operate expensive server infrastructure with many graphics cards can use Expert Offloading to run powerful models on more modest hardware while saving costs. Response times are then slower than with full operation in VRAM, but for tasks without strict real-time requirements – such as summarizing documents or internal search – this is often entirely sufficient.

Expert Offloading should be distinguished from the related concept of quantization, in which model weights are stored with lower precision to save memory space. Both techniques can be combined and complement each other well: quantization shrinks the model, while Expert Offloading cleverly distributes it across available memory tiers.