Memory Bandwidth

A computer chip does not compute with numbers it possesses itself. It fetches them from memory, processes them, and writes the result back. Memory bandwidth describes how much data can flow along this path per second. It is usually given in gigabytes or terabytes per second. A modern graphics chip for AI manages several terabytes per second, while a normal laptop achieves more like fifty to one hundred gigabytes. You can picture this like a highway between a warehouse and a workshop: the bandwidth is the number of lanes, not the size of the warehouse.

Why chips often wait for data

Computing power has grown much faster over the past decades than memory speed. This creates an imbalance: the compute units could work faster, but they don’t get enough supply. Experts call a chip in this situation memory-bound. It then doesn’t run at full performance but instead waits much of the time.

This exact case occurs very frequently with large language models. When such a model generates a single word, the chip must read through practically all stored model numbers once. For a model with 70 billion numbers, that amounts to several dozen gigabytes, depending on the storage format — for a single word. The computational steps themselves are comparatively cheap.

That is why, for AI accelerators, bandwidth is now advertised nearly as prominently as compute power. A chip with twice as many compute units but the same bandwidth delivers hardly any more speed for such tasks. This also explains why specialized high-speed memory has become one of the most expensive components in an AI server.

Lanes, clock rate, and stacked memory

Bandwidth results from two quantities. The first is the width of the data lines, meaning how many bits are transmitted simultaneously. The second is the clock rate, meaning how often a transfer occurs per second. Multiplying both values roughly gives the bandwidth. So you can either build more lanes or make the cars drive faster.

Continuously raising the clock rate runs into physical limits and costs a lot of power. The current approach therefore goes via width. With High Bandwidth Memory, or HBM for short, memory chips are stacked on top of each other and placed directly next to the compute chip. The short distances allow thousands of data lines in parallel. This is how the terabyte figures of modern AI accelerators come about.

A common misconception is confusing bandwidth with memory size. Size indicates how much fits in at all — it determines whether a model fits on the chip. Bandwidth indicates how quickly you can get to that data. Another separate value is latency: the waiting time until the first response from memory arrives. High bandwidth combined with high latency is possible, similar to a very wide but long tunnel.

From datasheets to the graphics card in your PC

In reports about new AI chips, bandwidth almost always appears in the first lines. When a manufacturer announces that its new accelerator achieves several terabytes per second, this is the value being referred to. Reports about scarce HBM memory also belong to this topic. Because only a few companies worldwide manufacture such memory, it is considered a bottleneck for the entire AI industry.

In everyday life, the term comes up when buying a graphics card or a smartphone. Two devices with the same amount of memory can differ significantly in bandwidth. Anyone running an AI model on their own computer notices this immediately in the output speed. The number of words per second there depends almost directly on the bandwidth.

Techniques for model compression can also be better understood this way. If each model number is stored with only 4 bits instead of 16 bits, the amount of data to be read shrinks to a quarter. The model then responds noticeably faster, even though it performs no fewer computational steps. The gain comes solely from the fact that less data has to pass through the bottleneck.