HBM3E

Every computer needs working memory. This is where the data that the processing chip is currently working with is stored. In normal PCs, these memory chips sit as elongated modules on the motherboard, a few centimeters away from the processing chip. HBM3E does things differently: here, eight to twelve thin memory chips are stacked on top of each other, and this tower is placed directly next to the processing chip on the same substrate. The abbreviation stands for High Bandwidth Memory, third generation, and the E stands for an improved development stage (“Extended”). The purpose of all this is that far more data per second can flow between memory and processing chip than with conventional memory.

The bottleneck of modern AI chips

A large language model consists of many billions of stored numbers, the so-called parameters. For every single word a chatbot outputs, the chip has to fetch a large portion of these numbers from memory. The actual computation often takes less time than fetching the data. Experts call this memory-bound: the processing chip waits instead of calculating.

You can picture it like a very fast checkout in a supermarket. If goods only arrive via a narrow conveyor belt, the fast checkout is useless. HBM3E massively widens this belt. A single stack achieves around 1.2 terabytes per second, whereas a typical PC memory module manages about 0.05 terabytes per second.

This is why HBM3E has become an economically sensitive component. Only three companies worldwide manufacture such memory in large quantities: SK Hynix, Samsung, and Micron. If their production runs short, the processing chips themselves cannot be delivered either. Reports of HBM capacity being sold out a full year in advance appear regularly in business news.

Chip stacks and through-silicon vias

The speed does not come from particularly fast individual memory cells. It comes from the sheer number of connections. A normal memory module transfers data over 64 parallel lines. An HBM3E stack uses 1024. Each individual line is actually rather slow, but the sheer quantity makes the difference.

This is made possible by two construction tricks. First, the stacked chips are drilled through vertically and the holes are filled with metal. These tiny connections are known in technical jargon as TSVs, Through-Silicon Vias. They connect all levels of the tower to each other without cables having to run around the outside. Second, the entire stack sits together with the processing chip on a shared silicon substrate, the interposer.

The price for this is high. The chips have to be ground down to extreme thinness so the stack doesn’t become too tall. If a single layer is defective, the entire stack becomes scrap. In addition, heat builds up inside the tower that is difficult to dissipate. This is exactly why HBM3E costs many times more per gigabyte than ordinary working memory and is not installed in normal computers.

Built into Nvidia accelerators and into quarterly earnings

Anyone wanting to buy HBM3E directly has hardly any chance to do so. The memory is not sold individually but is permanently built into an accelerator chip. You encounter it in data centers: in Nvidia’s H200 and Blackwell cards, in AMD's Instinct series, and in similar products. Anyone using a chatbot is indirectly using HBM3E without noticing it.

In financial news, however, the term comes up constantly. Memory manufacturers report HBM revenues separately in their quarterly results, because this business is significantly more profitable than classic memory. The question of who has passed qualification at Nvidia, i.e. been approved as a supplier, also moves stock prices.

A common misconception is that HBM3E makes a chip generally faster. This only applies to tasks with very large amounts of data. In gaming or office applications it brings almost no benefit, because there the data volumes are small enough for normal memory. The next stage is already called HBM4 and is meant to double the number of lines once again.