NVLink

Large AI programs are not processed by a single computing chip, but by hundreds or thousands at the same time. These chips must constantly exchange intermediate results with one another. This is exactly what the company Nvidia developed a proprietary cable and connector system for: NVLink. It connects graphics chips directly to one another, instead of taking the detour via the motherboard’s normal slots. The advantage is speed: several hundred gigabytes per second flow over NVLink, equivalent to the contents of many hard drives. Think of it as a private highway between two factory buildings, while everyone else takes the congested country road.

Why fast chip-to-chip links determine AI performance

Modern AI models are too large for a single chip. Their learned numerical values, the so-called parameters, require more memory than a single graphics chip possesses. That’s why the model is split up: each chip gets a piece. With every computation, the chips must combine their partial results.

If the connection is too slow in the process, the chips end up waiting for one another. They might then only be computing for half the time and standing idle the rest. This is precisely what’s called a bottleneck: it’s not the computing power that limits the system, but the data transport. Data centers cost billions, so every minute of idle time is expensive.

For Nvidia, NVLink is also a business advantage. The technology only works with Nvidia hardware. Anyone who builds an entire data center on it cannot simply swap out individual parts for competitors' chips. That’s why other manufacturers are working on open alternatives that are being jointly developed by several companies.

From cable to switch fabric

In the simplest case, NVLink connects two chips directly. Each connection consists of several lane pairs, over which data travels in both directions simultaneously. A chip has several such ports, which can be bundled together. The more bundles, the higher the transfer rate.

With eight or more chips, direct cabling becomes impractical because the number of cables required increases sharply. That’s why there is the NVSwitch, a switching chip. It works like a railway junction: each graphics chip has only one line to the node, and the node forwards the data to the correct destination. This way, every chip can talk to every other chip at full speed.

An important side effect: via NVLink, a chip is allowed to access the memory of another chip, almost as if it were its own. To the program, the group then looks like a single, very large computing chip. NVLink should be distinguished from InfiniBand or Ethernet. These networking technologies connect entire server enclosures over greater distances and are slower. NVLink operates within a single enclosure or a server rack.

NVLink in server racks and in quarterly earnings

You rarely encounter NVLink directly, since the technology is embedded in data centers. However, it is used in almost every request made to a large chatbot. Training such models also runs over groups of chips connected via NVLink. In Nvidia’s GB200-series systems, several dozen chips are combined into a single unit in this way.

In business news, the term comes up in discussions of Nvidia’s lead. Analysts often emphasize that competitors can build individual chips, but not as easily an entire connected system. Watch for phrases like NVLink domain: this refers to the number of chips wired together into a single unit.

A common misconception is that NVLink is a long-distance network cable. Its range is actually more in the realm of centimeters to a few meters. A second misconception: more NVLink doesn’t make a model smarter. It only shortens waiting times and thereby lowers costs.