AI Accelerator

An AI accelerator is a computer chip designed for one very specific kind of computational work. Programs that learn from examples and then generate text or images do almost nothing else internally: they multiply huge tables of numbers with each other and add up the results. That amounts to billions of identical computational steps per response. A computer’s ordinary main processor can do this too, but it’s an all-rounder and largely works through the steps one after another. An AI accelerator forgoes many other capabilities and, in exchange, computes thousands of these simple steps simultaneously. This makes it ten to a hundred times faster for this one task.

Why ChatGPT couldn’t run without these chips

Modern AI systems are only possible because chips like these exist. A large language model has hundreds of billions of adjustable numerical values, known as parameters. Setting these values on ordinary processors would take decades. With tens of thousands of accelerators working in parallel, it can be done in weeks.

The second reason is power. Data centers pay for energy, and AI queries run around the clock. A specialized chip performs the same calculation using a fraction of the power. With millions of queries per day, this determines whether a service is affordable at all.

That’s why these chips have become an economic power issue. The manufacturer Nvidia became one of the most valuable companies in the world because of them. Whoever can’t get accelerators can’t compete in AI. This is why the United States has restricted the export of the fastest models to China.

Many simple computing units instead of a few clever ones

A main processor might have eight to sixteen computing cores. Each one is flexible and can control complicated processes. An AI accelerator, by contrast, has thousands of very simple computing units. Each individual one can do almost nothing except multiply and add. But all of them work in lockstep on the same large table of numbers.

The comparison to a construction site fits well. A main processor is a highly skilled craftsman who can solve any task, but alone. An accelerator is a column of a thousand workers who can only stack bricks. For a brick house, the column is unbeatable. For anything else, it’s useless.

A second trick is economizing on precision. Instead of storing numbers with many decimal places, these chips often deliberately calculate coarsely. For AI this is usually sufficient, because what matters is rough tendencies, not exact results. Coarser numbers require less memory and less time. Also important is a very fast connection to memory, since the number tables must constantly be resupplied.

From the data center to the smartphone

The best-known form is the graphics card, or GPU for short. It was originally developed for video games, because image computation likewise consists of many similar steps. This very property makes it useful for AI. When news reports talk about AI chips, they usually mean data-center GPUs, such as Nvidia’s H100 or B200. A single such device costs several tens of thousands of euros.

Beyond that, there are true specialized chips with no graphics heritage. Google builds its own TPUs, while Apple and Qualcomm embed small accelerators directly into smartphone chips. These are often called Neural Engine or NPU. They ensure that speech recognition, photo enhancement, or translation run directly on the device, without an internet connection.

A common misconception is that an AI accelerator is simply an especially fast processor. It is not. It doesn’t replace the main processor, but works alongside it. The main processor coordinates and forwards the computation-heavy parts to the accelerator. Together, the two form an AI-capable system.