Memory Bottleneck

A computer chip has to do two things: compute, and fetch the numbers it computes with. These numbers sit in memory, in components next to the actual compute unit. But the path from memory to the compute unit has a limited width. If the chip could compute faster than the supply arrives, this is called a memory bottleneck. The chip isn’t too slow then, it’s underserved. A picture for this: a kitchen with twenty cooks, but only one narrow door through which ingredients can come in.

Why expensive chips spend most of their time waiting

Over the past decades, chip compute power has grown much faster than memory speed. A modern graphics chip can perform roughly a hundred times more compute operations per second than it can reload numbers. For many tasks, this means: the compute unit sits idle and waits. Experts also call this gap the Memory Wall.

For language models, this is especially painful. When a chatbot generates one word after another, it has to read through all of the model’s stored values once for every single word. For a large model, that quickly adds up to hundreds of gigabytes of data per word. The actual computation involved is comparatively tiny. Response speed therefore depends almost entirely on memory bandwidth, that is, on how much data per second can pass through the connection.

Economically, this explains a lot. Anyone renting compute time in a data center pays for the time the chip is occupied, not for the compute operations actually performed. A chip that sits idle eighty percent of the time still costs the full price. That’s why memory has become one of the most important selling points for AI hardware.

Bandwidth, cache, and the tricks against it

Memory is structured in tiers, similar to a desk with a shelf and a basement. Very close to the compute unit sits a very small, very fast intermediate storage, the cache. Behind that lies the large main memory of the graphics card, slower but much roomier. The further out the data sits, the longer access takes. The bottleneck arises wherever data has to cross one of these boundaries.

A simple measure for this is arithmetic intensity: how many compute operations a program performs per byte loaded. If this value is high, loading pays off, and the chip stays busy. If it’s low, memory access dominates. Experts then say the program is memory bound rather than compute bound.

There are several countermeasures. You can store the model’s numbers more coarsely, for example with eight instead of sixteen bits per value; this halves the amount of data that needs to be loaded. You can batch multiple requests together, so that a value loaded once benefits many users at the same time. And you can restructure computation steps so that intermediate results stay in the cache instead of migrating to main memory. The best-known representative of this idea in the AI field is called FlashAttention.

HBM, Nvidia, and the headlines about it

When business news talks about HBM, this is exactly the problem at issue. HBM stands for High Bandwidth Memory, meaning memory with an especially wide connection. The chips are stacked and mounted directly next to the compute unit, instead of being connected via long circuit traces. Manufacturers like SK Hynix, Samsung, and Micron earn substantial sums from this, and at times HBM was scarcer than the compute chips themselves.

The topic also comes up in product announcements. When a new AI accelerator is marketed, alongside compute power there are almost always figures for memory size in gigabytes and bandwidth in terabytes per second. For running language models, the second figure often says more about real-world speed than the first.

A common misconception is that more memory automatically means faster. More capacity allows for larger models, but doesn’t speed anything up as long as the connection stays the same width. Capacity and bandwidth are two different things. Anyone who looks only at the gigabytes on a spec sheet is missing exactly the bottleneck this is all about.