On-Chip Memory

A computer chip consists of two kinds of components: those that compute, and those that hold numbers. On-chip memory is the memory that sits directly on the same die as the compute units. It is therefore only fractions of a millimeter away from the spot where the actual computation happens. Its counterpart is memory that sits as a separate component next to it on the board and is connected via wires. The difference sounds like a minor detail, but it determines how fast a chip can operate. Because fetching data over a long connection costs time and power — both to a surprising degree.

Why short distances determine speed

Compute units have become extremely fast today, while memory connections have lagged far behind. A modern compute unit can perform hundreds of operations in the time it takes to access external memory. Experts call this mismatch the memory wall. The chip isn’t too slow — it’s simply waiting constantly for more data.

Then there’s power consumption. Fetching a number from on-chip memory costs roughly a hundred times less energy than loading the same number from an external memory chip. In a data center answering millions of AI requests per day, this adds up to real electricity costs. That’s why memory architecture isn’t a niche detail for specialists, but a topic that shows up in quarterly reports.

The price for this is size. On-chip memory takes up area on the silicon, and silicon area is the most expensive real estate on the entire chip. A graphics processor might have a few hundred megabytes of it, while tens of gigabytes of external memory sit right beside it. So you end up with very fast space in tiny quantities and slow space in abundance.

A layered structure

Chips resolve this dilemma with a hierarchy. Closest to the compute unit sit tiny registers, followed by several levels of so-called caches, known as L1, L2, and L3. The farther out, the larger and slower. Furthest out sits the external main memory.

Think of it like a desk. What you’re currently holding in your hand are the registers. The stack right in front of you is the cache, and the shelf on the wall is main memory. A good chip ensures that the data currently needed is already sitting on the stack as often as possible. When that succeeds, it’s called a hit; otherwise, a miss.

Technically, on-chip memory usually consists of SRAM. This type of memory needs six transistors per stored bit and is therefore very area-hungry. External memory, by contrast, uses DRAM, which gets by with one transistor and a tiny capacitor. DRAM is cheaper and more densely packed, but it must refresh its contents thousands of times per second — which makes it slower.

Why AI chips are measured by this

Large language models consist of billions of stored numbers, the parameters. For every answer, a chip must work through these numbers. They never fit entirely into on-chip memory, so they must constantly be reloaded. This is exactly where it’s decided whether an expensive accelerator card actually reaches its theoretical performance.

That’s why the term regularly appears in product announcements. AI chip developer Cerebras advertises having its entire memory on the chip, rather than next to it. Other manufacturers stack external memory directly next to the compute chip and connect it with an extremely wide interface — a technique called HBM, a compromise between both worlds.

A common misconception is that more on-chip memory is always better. Every additional megabyte displaces compute units from the same area. Chip design therefore consists of trade-offs, not maximization. The same compromise exists in smartphones and laptops too, just with different numbers.