GPU

Every computer contains a main processor. It works through tasks very quickly one after another, but only a few at a time. A GPU is a second chip, built in a completely different way. It consists of thousands of small computing units that all work at the same moment. Each individual one is slow and can only handle simple tasks. Together, however, they manage enormous volumes of simple calculations. The abbreviation stands for Graphics Processing Unit, in other words graphics processor.

Why GPUs became the bottleneck of the AI industry

At their core, AI systems calculate almost exclusively with lists and tables of numbers. Millions of such values are multiplied and added. These calculations barely depend on one another, so they can be carried out simultaneously. This is exactly what a GPU is built for. On a main processor, training a large language model would take decades. On thousands of GPUs, it takes weeks.

This has created an economic bottleneck. The US manufacturer Nvidia dominates this market and has thereby become one of the most valuable companies in the world. Individual specialized cards for data centers cost between 20,000 and 40,000 dollars depending on the model. Large AI companies buy tens of thousands of them. Anyone who can't get any cannot compete in developing large models.

That is why GPUs are also politically important. The US bans the export of its fastest AI chips to China. Computing power is thus being treated like a strategic resource. Reports about delivery times, export rules, or new chip generations regularly move the stock markets.

Many slow computing units instead of a few fast ones

The difference can be compared to two ways of working. A main processor is like a single, very experienced craftsman. He can also solve complicated tasks, but only one after another. A GPU is like a hall with a thousand helpers, all performing the same simple hand movement. For a thousand identical parts, the hall is unbeatable. For a single tricky repair, it's useless.

For this to work, the task must be divisible. With images, this is obvious: each pixel is calculated independently. The same applies to AI models for the many multiplications in a table of numbers. Tasks in which each step waits for the result of the previous one, however, bring no advantage.

A second point is memory. A GPU has its own, very fast connected memory directly on the card. However, it is limited, often 80 to 192 gigabytes per chip. A large model therefore does not fit on a single GPU. It is spread across many cards that work together via fast connections. Often it is not the computing power that is the bottleneck, but rather reloading the data into memory.

From game console to data center

The first GPU in most people's lives is found in a gaming device. In consoles, gaming PCs, and even in every smartphone, a graphics processor calculates the images on the display. These chips are smaller and more power-efficient than the data center variants, but work on the same principle. Video editing, 3D software, and image processing also use them.

Anyone working with AI themselves usually rents GPUs rather than buying them. Cloud providers rent out computing time by the hour, often for just a few euros for small cards. When chatting with an AI assistant, a GPU is also running in the background in a data center. All you notice is the one-second wait time.

Related terms also appear in the news alongside this. Google's TPUs and other AI accelerators are even more specifically tailored to AI calculations and can do fewer different things. A common misconception is that a GPU is fast on its own. Its advantage only arises with tasks that can be broken down into many similar parts.