High-Bandwidth Memory

Every computer chip that performs calculations needs a place to hold the numbers it is currently working with. This intermediate storage is called working memory. In normal computers, it sits as a small circuit board a few centimeters away from the compute chip. High-Bandwidth Memory does it differently: the memory consists of several thin memory dies stacked on top of each other and glued directly onto the compute chip. Because the distance is so short, a very large number of data lines fit in parallel between them. As a result, an enormous amount of data reaches the chip every second. That is exactly what the word bandwidth means: the amount of data that flows through a connection per second.

Why AI chips starve without HBM

An AI model consists of many billions of stored numbers, the parameters. For the model to generate a response, the chip must fetch these numbers from memory and compute with them. This doesn’t happen just once, but anew for every single word of the response. So the chip is constantly reading gigantic amounts of data.

This creates an imbalance. The compute units of modern graphics chips have become so fast that they are mostly waiting. It isn’t the computing that slows things down, but the supply of data. Experts call this memory-bound: performance depends on bandwidth, not on computing power. A race car on a single-lane road is just as fast as a compact car.

That’s why HBM helps determine what an AI data center can achieve. The memory is also an economic factor: in expensive AI accelerators, the HBM installed accounts for a significant share of manufacturing costs. Worldwide, only a few companies can produce it, chiefly SK Hynix, Samsung, and Micron. If their production can’t keep up, finished AI chips are missing — even if there were enough compute chips available.

Stacks, through-silicon vias, and a shared carrier substrate

An HBM module consists of four to sixteen memory chips stacked on top of one another. They are connected by tiny holes drilled vertically through the silicon and filled with metal. These through-silicon vias replace long circuit traces. As a result, the data path runs up and down instead of across a circuit board.

The stack is then placed together with the compute chip onto a shared carrier substrate, known as the interposer. Thousands of fine lines run across this carrier between the memory and the chip. Normal graphics memory has a few hundred such lines, HBM has several thousand. Each individual line isn’t particularly fast on its own — the sheer number makes the difference. That’s the trick: many slow lanes instead of one fast one.

The price for this is complexity. Stacking and connecting the dies is delicate, and a single defective chip in the stack renders the whole package unusable. As a result, production yield is lower than for ordinary memory. Another common misconception is that HBM is simply more memory. It is, above all, faster memory; capacity per module is often only a few dozen gigabytes.

HBM in products and quarterly earnings

HBM is most commonly mentioned in connection with AI accelerators for data centers. Nvidia's chips for AI training carry HBM, as do competing products from AMD and the in-house chips developed by major cloud providers. When a company unveils a new chip generation, the HBM version is almost always among the specifications cited, such as HBM3e or HBM4.

In business news, the term comes up for a different reason. Memory manufacturers earn considerably more from HBM than from memory for phones and laptops. Reports of HBM capacity being sold out for years to come therefore move stock prices. Supply shortages for AI chips are also often attributed to memory and packaging technology rather than to the compute chips themselves.

In one’s own devices, however, one practically never encounters HBM. Gaming graphics cards use the cheaper GDDR memory, laptops use ordinary DDR modules. But anyone using a chatbot is using HBM indirectly: the response is generated in a data center, on a chip whose memory stack holds the model’s parameters ready.