Amdahl's Law

Some tasks can be split up, others cannot. Someone painting a wall will finish much faster with ten helpers. Someone who has to read a book and then summarize it gains almost nothing from ten helpers, because the steps follow one another. Amdahl’s Law is the mathematical formula behind this observation. It states: the portion of a task that must necessarily be done in sequence sets a hard upper limit on the possible speedup. The computer scientist Gene Amdahl formulated this idea in 1967, just as computers with multiple processing units were emerging.

Why more chips don’t automatically mean more speed

The numbers are unforgiving. Let’s assume 90 percent of a task can be distributed across any number of processing units. The remaining 10 percent must strictly run in sequence. Even with infinitely many processors, the whole thing then only becomes ten times faster. The ten percent remains and determines the outcome.

This is exactly where many expectations of modern hardware fail. A data center with ten thousand graphics cards sounds impressive. But if a training process gets stuck at a single point where all cards must wait for a shared intermediate result, sheer quantity hardly helps. Engineers then speak of a bottleneck, meaning a point through which everything must pass.

Amdahl’s Law also explains why a computer with eight cores — that is, eight processing units in one chip — doesn’t feel eight times faster in everyday use. Starting up a program, loading a file, waiting for a response from the network: all of this runs sequentially, that is, step by step. These portions are not shortened by additional cores.

The formula behind the upper limit

One can conceptually divide the work into two buckets. One portion is parallelizable, meaning it can be handled simultaneously by several units. The rest is sequential and stays the same length, no matter how much hardware is added. The total time is the sum of the unchangeable part and the shared part.

Working through the math, the maximum speedup turns out to be one divided by the sequential portion. At 10 percent sequential portion, that’s a factor of 10. At 1 percent, it’s a factor of 100. At 50 percent, it’s only a factor of 2, even with a million processors. The decisive lever, then, lies not in the amount of hardware, but in shrinking the sequential portion.

On top of that, in practice there is additional overhead that the formula doesn’t account for at all. Processing units must coordinate, exchange data, and wait for one another. This administrative overhead grows with the number of participants. That’s why some programs actually become slower again beyond a certain number of cores. A related idea is Gustafson’s Law: it points out that with more hardware, people in practice usually tackle bigger problems rather than solving the same problem faster. Then the balance looks more favorable.

From graphics cards to project planning

In tech news, the law usually comes up in connection with AI data centers. When a provider reports that a model training run only benefits by 80 percent from double the hardware, Amdahl is behind it. It’s also the reason why manufacturers of processors for phones and laptops don’t endlessly add cores, but instead make individual cores faster.

A common misconception is that the law is an argument against parallel hardware. The opposite is true: it only says where one must intervene. Anyone wanting to speed up an application first looks for the sequential bottleneck and eliminates it. Only after that does additional hardware pay off.

The idea can even be applied to teamwork. A software project doesn’t become arbitrarily faster with more developers, because coordination and dependencies remain. Anyone who has ever waited in a group project for one person’s result knows Amdahl’s Law from personal experience.