InfiniBand

A modern data center consists of thousands of individual computers housed in racks. These computers must constantly exchange data with one another. InfiniBand is a technology that does exactly that: cables, add-in cards, and switches that connect the machines to each other. It is an alternative to Ethernet, the networking technology commonly used in offices and homes. The difference lies in speed and, above all, in latency. InfiniBand is designed so that data arrives in millionths of a second, not merely thousandths of a second.

Why AI training fails without fast connections

Large AI models have long since stopped fitting into a single computer. The model is therefore distributed across thousands of graphics cards that compute in parallel. After every computation step, these cards must synchronize their intermediate results. This doesn’t happen once a day, but many times per second.

As a result, the network becomes the bottleneck. If the connection is too slow, expensive graphics cards sit idle waiting for data. Experts refer to this as poor utilization. In a training run that takes weeks and costs millions, this makes an enormous difference.

A comparison helps: group work rarely fails because individuals think too slowly. It fails because coordinating with one another takes too long. It is precisely this coordination that InfiniBand speeds up.

Direct access to someone else’s memory

The most important principle behind InfiniBand is called RDMA, short for Remote Direct Memory Access. With it, one computer writes data directly into another computer’s working memory. The target machine’s operating system is not involved at all. In normal networks, by contrast, every data packet has to travel through several software layers, which costs time.

Think of it like package delivery. Normally, the courier rings the bell, someone opens the door, sorts the package, and places it on the shelf. With RDMA, the courier puts the package directly into the correct slot themselves, because they have the key. This shortcut lowers latency to about one microsecond, i.e., one millionth of a second.

On top of that comes a property called lossless transport. Ethernet discards data packets if a device becomes overloaded and has them resent. InfiniBand instead throttles the sender beforehand, so that nothing is lost. Current generations transmit 400 gigabits per second and more over a single link.

Nvidia, Mellanox, and the dispute with Ethernet

InfiniBand originally came from the Israeli company Mellanox. Nvidia bought it in 2019 for around 6.9 billion dollars. As a result, Nvidia today sells not just the graphics cards for AI, but also the networking technology in between. In quarterly reports, this segment appears as its own line item and is growing strongly.

Anyone who reads news about new AI data centers will regularly come across this term. It usually concerns the question of how many graphics cards can be sensibly interconnected. InfiniBand has also frequently appeared for years in the rankings of the world’s fastest supercomputers.

A common misconception is that InfiniBand has no alternative. Competitors such as Broadcom and an industry alliance called the Ultra Ethernet Consortium are working to make Ethernet similarly fast. Major operators like Meta are already relying on such Ethernet variants. In home Wi-Fi, however, InfiniBand will never show up: the technology is too expensive and intended only for short distances within the data center.