CUDA Core

A graphics card is an add-on component in a computer that was originally intended for images in games. Nvidia's cards contain thousands of tiny computing units that Nvidia calls CUDA cores. Each individual one can only perform very simple tasks: add two numbers, multiply two numbers. The strength lies in the quantity. While a computer’s main processor might have eight powerful units, a modern graphics card brings over 10,000 of these small cores. They all work simultaneously on the same problem.

Why thousands of small cores made AI possible in the first place

An AI model learns by computing enormous tables of numbers against each other. These calculations are simple individually, but there are an unimaginable number of them. This is exactly what the architecture of a graphics card is ideal for. A normal processor would work through the tasks one after another and would need years. Thousands of CUDA cores share the work and accomplish the same thing in days.

A comparison makes this vivid. The main processor is like four professors solving difficult, different tasks. The graphics card is like ten thousand students who all receive the same simple calculation task with different numbers. For a crossword puzzle, the students are useless. For ten thousand multiplications, they are unbeatable.

This property also explains Nvidia’s economic rise. The company built cards for computer games for decades. When it became clear that AI needed the same kind of computing power, Nvidia became one of the most valuable companies in the world. The CUDA core is the smallest building block behind this success.

How the work is distributed across the cores

CUDA is not just a component, but also a programming system from Nvidia. A programmer writes a short calculation instruction and specifies how many times it should be executed. The graphics card then distributes these tasks itself among its cores. The cores are organized into groups and work in lockstep.

This lockstep operation is both a strength and a weakness. All cores in a group execute the same instruction at the same moment, only with different data. As soon as a program contains many branches, meaning many if-then decisions, the speed collapses. Part of the cores then has to wait until the other part is finished. That’s why graphics cards are not suited to every kind of software.

A common misconception is to read the number of CUDA cores directly as a measure of performance. Two cards with the same core count can have different speeds, because clock frequency and memory connection also play a role. Often the memory is even the bottleneck. The cores are of no use if the supply of numbers arrives too slowly. Modern cards also contain Tensor cores, specialized units solely for the matrix calculations of AI. These are to be distinguished from CUDA cores.

Where the core count appears in datasheets and headlines

The term is encountered most directly when buying a graphics card. In the datasheet of a GeForce card, the number of CUDA cores is listed as one of the first specifications. There it serves as a rough guide to how much computing power a card offers within the same product line. Between different generations, however, the comparison is of little informative value.

In business news, the term comes up when it concerns data centers. Companies like OpenAI or Meta buy tens of thousands of Nvidia cards for their AI systems. Export restrictions on chips to China also revolve around exactly this computing power. Anyone who wants to understand the headlines should know which component is behind them.

Indirectly, anyone who uses a chatbot benefits from this. Every answer is generated on servers whose graphics cards perform the necessary calculations in parallel. Video editing, weather forecasts, and scientific simulations also run on CUDA today. The term thus stands for far more than just computer games.