CUDA

A graphics card was originally intended only for images: for games, videos, 3D rendering. But there is one thing it can do especially well, namely carry out thousands of simple calculations simultaneously. CUDA is a collection of programs and tools from the company Nvidia that opens up exactly this capability for arbitrary computing tasks. The name stands for Compute Unified Device Architecture, though the full name says almost nothing and is barely ever used, even within the industry. What matters is this: with CUDA, a developer writes code that does not run on a computer’s normal main processor, but on the graphics card. Nvidia introduced CUDA in 2007, long before today’s AI boom began.

Why this gives Nvidia ownership of the AI market

Modern AI models consist at their core of enormous number tables that are multiplied together. This exact type of calculation can be broken down well into thousands of small subtasks. Depending on the task, a graphics card handles this ten to a hundred times faster than a main processor. Without this leap in speed, training today’s language models would be practically impossible.

Nvidia’s real advantage, however, does not lie in the hardware alone. Over almost two decades, a massive ecosystem has formed around CUDA. Almost every AI tool that researchers use requires CUDA. Anyone who buys a graphics card from another manufacturer often has to adapt software from scratch or forgo certain libraries.

In economic terms, this is called a moat: a lead that competitors can only catch up with great difficulty. AMD and Intel do build competitive chips. But their software environments are considered less mature. This explains a large part of Nvidia’s stock market value, which crossed the three-trillion-dollar mark for the first time in 2024.

What happens when computing on the graphics card

CUDA consists of three parts. First, an extension of the C programming language used to write graphics-card code. Second, a driver that sends this code to the card. Third, ready-made libraries for common tasks such as matrix multiplication or image processing.

A developer writes a small function, called a kernel in CUDA jargon. This function is not executed once, but thousands of times in parallel. Each copy processes a different data point, such as a single pixel or a number in a table. The graphics card itself distributes these copies across its computing units, the so-called CUDA cores. A current card has several thousand of these, while a main processor has eight to sixteen cores.

The common misconception: graphics cards are not generally faster. For a single long chain of calculations, where each step builds on the previous one, a main processor is superior. The advantage only arises when many similar calculations can be performed independently of one another. A second stumbling block is data transfer, since copying between main memory and graphics card memory costs time.

Where the name appears in news and products

CUDA itself is rarely visible directly, since hardly anyone programs with it by hand. Instead, developers use tools such as PyTorch or TensorFlow, ready-made program toolkits for AI. These access CUDA in the background. However, anyone who tries to run an AI model on their own computer quickly runs into error messages about missing CUDA versions.

In business news, the term usually comes up in connection with Nvidia’s market power. When alternatives to Nvidia are reported on, the discussion almost always revolves around whether a competing piece of software can match CUDA. AMD’s counterpart is called ROCm, and an open approach backed by several companies is called OpenCL.

CUDA also plays a role in politics. Export restrictions on AI chips to China affect not only the hardware but also access to this software environment. And anyone renting cloud computing time from Amazon, Google, or Microsoft is typically paying for machines with Nvidia cards and pre-installed CUDA.