CUDA Cores

A graphics card is a separate computing component in the computer that was originally intended for rendering images in games. It contains thousands of tiny computing units, all working at the same time. On cards made by the company Nvidia, these computing units are called CUDA cores. Each individual one can only do very simple things: add two numbers, multiply two numbers. Its strength lies not in capability, but in quantity. A current gaming graphics card has between 4,000 and 20,000 such cores. The name CUDA comes from the software platform through which Nvidia makes these cores accessible to programmers.

Why graphics cards are driving the AI industry

The normal main processor of a computer, the CPU, usually has between 8 and 32 cores. These cores are individually very fast and very flexible. They can make complicated decisions and process programs with many branches. For tasks where one step follows another, the CPU is unbeatable.

When training and running AI models, however, the work looks completely different. There, huge tables of numbers are multiplied with each other, millions of times in a row. The individual calculations do not depend on one another. They can therefore all be executed at the same time. A wall of thousands of CUDA cores is ideal for exactly this.

This is why Nvidia became one of the most valuable companies in the world within just a few years. The lead is not only due to the hardware. Because almost all AI programs have been written for CUDA over the years, switching to chips from other manufacturers is costly. This binding to a single platform is called the lock-in effect.

A thousand calculators for the same task

An analogy helps here. The CPU is a single professor who solves a difficult problem in many steps. The graphics card is a hall with a thousand students who all only know basic arithmetic. For a tricky proof, the professor is better. For ten million simple multiplications, the hall is many times faster.

Technically, the CUDA cores work in fixed groups. All cores in a group execute the same instruction in the same clock cycle, just on different data. This principle is called SIMD, meaning one instruction on many pieces of data at the same time. It also explains the weakness of this design: as soon as a program contains many if-then branches, parts of the group have to wait. The computing power goes to waste.

An important distinction is often overlooked in spec sheets. Alongside the CUDA cores, modern Nvidia cards also contain so-called Tensor cores. These are specialists for exactly one task: multiplying small blocks of numbers, as constantly occurs in AI models. The majority of AI computing power today comes from the Tensor cores, not from the CUDA cores. The number of CUDA cores is therefore not a reliable measure of AI speed.

From spec sheets to the data center

You most often encounter this figure when buying a graphics card. There it appears in the spec sheet next to memory size and clock speed. A common mistake is to directly compare the figures from different manufacturers. AMD calls its computing units stream processors, and Intel calls them something else entirely. The numbers are not equivalent, because the cores are built differently.

In business news, the term comes up when data centers are discussed. Large AI providers buy tens of thousands of specialized cards such as the Nvidia H100 or B200. US export restrictions against China also revolve around such chips. Whenever computing power is mentioned in this context, it is almost always this parallel design behind it.

And finally, you notice the cores without seeing them. Every smoothly rendered game image, every rendered video, and every chatbot response is produced on hardware like this. Only the path to getting there is now different from rendering game graphics.