Neural Accelerator

Every computer contains a main processor. It is an all-rounder: it can process text, copy files, control the operating system. Artificial intelligence programs, however, pose a very one-sided demand. They multiply and add gigantic tables of numbers, over and over, billions of times. A neural accelerator is a chip that does exactly this one task extremely well and almost nothing else. That is why it is also called an AI accelerator or, when it sits inside phones and laptops, an NPU, short for Neural Processing Unit.

Why specialized chips are carrying the AI wave

An all-rounder processor is surprisingly inefficient at AI calculations. It tends to work through tasks one after another and spends a lot of time shuffling numbers back and forth between memory and the computing unit. A neural accelerator, by contrast, computes thousands of values simultaneously. For the same task, it often needs only a fraction of the time and energy.

Energy is the decisive factor here. Large data centers for AI consume as much electricity as smaller cities. Every watt saved per computing operation translates directly into money here. On the other side stands the smartphone: its battery lasts only a few hours. Without an energy-efficient specialized chip, live translation or facial recognition would hardly be possible on it.

Economically, this has turned the chip industry upside down. Companies that build or design such accelerators are now among the most valuable companies in the world. They regularly appear in stock market news, because their delivery times determine how quickly other companies can launch their AI products.

Matrices, number formats, and short paths

The core is a type of computation called matrix multiplication. A matrix is simply a table of numbers. A neural network, the computational model behind modern AI, essentially consists of such tables. An answer is produced by processing input values layer by layer against these tables. A neural accelerator contains grids of thousands of tiny computing units that carry out these multiplications in parallel.

A second trick is coarser number formats. A normal processor calculates very precisely, with many decimal places. For AI, a much coarser precision is often sufficient, for example eight instead of thirty-two bits in memory. The result barely gets worse, but the chip becomes much faster. Experts call this coarsening quantization.

The third point is short paths. Fetching data from main memory costs more energy than the calculation itself. Neural accelerators therefore have very fast memory located directly next to the computing units. You can picture it like a workshop where the tools are within arm's reach instead of down in the basement.

From phone chip to data center

You are probably already carrying a neural accelerator in your pocket. Almost all current smartphone chips contain an NPU. It sorts photos by faces, removes background noise during calls, and lets voice assistants respond without data traveling to the internet. Newer Windows laptops also advertise an NPU, often marketed under the label AI PC.

In data centers, larger variants dominate. Graphics cards were originally designed for video games, but they also compute in a highly parallel fashion and thus became the standard tool for AI training. Alongside them, there are pure AI chips such as Google's TPU. The line between graphics card and neural accelerator is blurring, because modern graphics cards contain dedicated units solely for matrix computations.

A common misconception is that such a chip is inherently intelligent on its own. It is merely a calculating machine. Without a trained model and matching software, it lies idle. Conversely: when news reports mention export restrictions on AI chips or billion-dollar investments in data centers, they are almost always talking about neural accelerators.