Von Neumann Architecture
The von Neumann architecture is the basic blueprint of almost all computers: the processing unit and memory are separate, and both data and the program itself reside in the same memory. Because everything has to run over a shared connection, a bottleneck arises that becomes especially noticeable in AI applications.
Almost every computer consists at its core of two parts. One part calculates, the other remembers things. The calculating part is called the processor, the remembering part is called memory. The von Neumann architecture is the blueprint that separates these two parts and connects them via a shared line. Its most important trick: the same memory holds not only the numbers used for calculations, but also the instructions themselves. It is named after the mathematician John von Neumann, who described it in 1945 in a widely read report.
Why this made a computer versatile
Before this idea, calculating machines were fixed to a single task. Anyone wanting to compute something different had to rewire cables or reset switches. This took hours or days. A computer was thus more of a specialized tool than a device for many purposes.
Because the program now sits in memory like any other number, this changes fundamentally. You simply load a different program, and the same machine does something completely different. That is exactly why your laptop can play videos, write essays, and render games without any change to the hardware. This versatility is the reason this blueprint has become established worldwide.
There is a second consequence that is less obvious. A program can treat another program as data, meaning it can modify or generate it. This is the basis for compiler programs, which convert source code into machine instructions. Without this property, modern software development would be barely conceivable.
The path of a single instruction
A von Neumann computer works in an ever-recurring cycle. First, the processor fetches the next instruction from memory. Then it decodes what the instruction means. Next, it executes it and writes the result back. After that, the cycle starts over again, billions of times per second.
A counter keeps track of where in memory the next instruction is located. Jump instructions can change this counter. This is precisely what gives rise to loops and branches, that is, repetitions and if-then decisions. In principle, a computer needs nothing more than this to execute arbitrary programs.
The catch lies in the shared line between processor and memory. Instructions and data must share this one path. Over the decades, processors have become much faster than memory access. This bottleneck is called the von Neumann bottleneck. As a countermeasure, small, very fast intermediate memories are placed directly next to the processor, so-called caches.
What this means for AI chips
An AI model consists of billions of stored numbers, the parameters. For every response, many of these must be fetched from memory. The actual calculation is often the smaller part of the effort. The bottleneck therefore plays a role in determining how fast and how expensive an AI response is.
That is why chip manufacturers today advertise less with computing power and more with memory bandwidth. Graphics cards for AI use very wide connections and memory that is built directly onto the chip. Some companies go further and build chips where computation happens right where the data is located. Such approaches are referred to as in-memory computing.
A common misconception is that these chips have abolished the von Neumann architecture. That is not true. They ease the bottleneck but still follow the same underlying principle. Anyone reading in financial news that memory manufacturers are benefiting from the AI boom is seeing this old blueprint from 1945 directly at work in current figures.