HBM
HBM (High Bandwidth Memory) is an especially fast type of memory that sits directly next to the compute chip and is used above all in AI accelerators such as GPUs. It is the decisive bottleneck for large AI models: whoever has faster HBM can process more data per second – and thus run larger models more efficiently.
HBM stands for High Bandwidth Memory, meaning roughly “memory with high transfer width.” It is a special form of memory that doesn’t sit somewhere on a computer’s main board, but directly next to the compute chip – sometimes even on the same small carrier. This allows extremely large amounts of data to flow simultaneously, far more than with conventional memory. HBM was originally developed for graphics cards, but today it is above all indispensable in AI chips. Without it, large language models like GPT or Gemini would be much slower in operation.
HBM as the bottleneck in the AI chip
An AI model consists of billions of numerical values, so-called parameters. With every request – for example, when you ask the chatbot a question – these values must be loaded from memory into the compute core. If this happens slowly, the compute core sits idle. It’s like a factory worker spending most of their time waiting for the next part instead of actually working.
This is exactly where the bottleneck lies: it’s not the chip’s computing speed that’s the problem, but how quickly data can be delivered from memory. This bottleneck even has a name – researchers call it the “memory wall.” HBM was developed to tear down this wall as far as possible. The currently widespread HBM3 generation achieves transfer rates of over one terabyte per second – equivalent to the contents of about 200 DVDs being shuttled back and forth every second.
How HBM achieves such high throughput
Normal memory – for example, the DDR5 memory in a laptop – is connected to the processor via a kind of data road. This road has a limited width, meaning a limited number of lanes through which bits can flow simultaneously. HBM solves this differently: multiple memory chips are stacked on top of each other, similar to floors in a building. Between the floors run thousands of tiny vertical connections, so-called Through-Silicon Vias (TSVs). This creates not a single narrow road, but thousands of parallel channels.
This stack is then placed directly next to the compute chip on a shared carrier called an interposer. The short distance significantly reduces the energy consumption per byte of data transferred. HBM is therefore not only faster than conventional memory, but also more efficient – an important point, since AI data centers consume enormous amounts of electricity. The downside: HBM is considerably more expensive to manufacture and harder to produce than normal memory.
Because the stacks are very complex to manufacture, there are only a few manufacturers worldwide capable of producing HBM – chiefly SK Hynix, Samsung, and Micron. This makes HBM a strategically scarce commodity.
HBM in products, news, and geopolitics
Anyone reading reports about AI chips will almost inevitably come across HBM. Nvidia's H100 and H200 series graphics processors, which form the backbone of most AI data centers worldwide, are equipped with multiple HBM stacks. The H100, for example, carries 80 gigabytes of HBM3 and thereby achieves a memory bandwidth of around 3.35 terabytes per second. Similar chips come from AMD and Google.
HBM also plays a growing role in geopolitics. The United States has introduced export restrictions on advanced AI chips to China – and these restrictions explicitly cover HBM memory as well, since it is considered essential for high-performance AI. Whoever controls HBM production thus holds a strategic lever in global AI development.
For consumers, HBM is invisible – it’s found in server farms, not in smartphones or laptops. Nevertheless, it has a direct impact on the response speed of AI services and on the model size that can be run at all in a meaningful way. Every answer a large language model gives has HBM somewhere in its supply chain.