Interconnect

Large AI systems are not computed on a single compute chip, but on thousands simultaneously. For these chips to work together on the same task, they must constantly exchange intermediate results. The cables, connectors, switches, and protocols needed for this are collectively called the interconnect. You can think of it as the road network of a data center: the chips are the factories, the interconnect is everything that moves goods between them. An interconnect is measured by two metrics. Bandwidth indicates how much data fits through per second; latency indicates how long a single message takes to reach its destination.

Why the fastest chip alone is of no use

When training an AI model, each chip only handles part of the work. After every computation step, all chips must pool and reconcile their results. This reconciliation doesn’t happen once a day, but many times per second. If the connection is too slow, the expensive chips just sit idle. You end up paying for compute power you’re not actually using.

In practice, this is a hard limit. In very large training runs, communication can consume a substantial share of the total runtime. A facility with 20,000 chips therefore rarely delivers twenty-thousand times the performance of a single chip. How close one gets to this ideal depends heavily on the interconnect. Experts refer to this as scaling efficiency.

That’s why the interconnect has become its own business field. Nvidia specifically secured this technology by acquiring the network hardware maker Mellanox. Broadcom, Cisco, and several startups also earn money from networking chips for AI data centers. Anyone reading stock market news will regularly come across the term there.

From copper cables in the rack to fiber optics between buildings

Interconnects exist on multiple levels, and they get slower the farther out you go. Directly on a circuit board or within a server rack, short copper lines connect eight or sixteen chips very tightly. Nvidia calls this technology NVLink. Such connections are extremely fast, but only possible over a few meters.

Between server racks, networking technology such as InfiniBand or very fast Ethernet takes over, usually via fiber optics. Here, data travels through switches, i.e., distribution nodes that forward messages. How these nodes are interconnected is called topology. A common design is the fat tree: many chips at the bottom, ever-thicker connections toward the top, so that no path becomes a bottleneck.

A common misconception is to equate interconnect with the regular internet. The difference is one of scale. A good home connection might manage one gigabit per second. Modern chip-to-chip connections operate in the range of hundreds of gigabits or several terabits per second. And they are designed for uniform, predictable delay, because a single slow chip drags down the entire group.

The term in product announcements and stock market news

With every new chip generation, the interconnect gets specifically promoted. Nvidia doesn’t present its systems as individual graphics processors, but as racks in which dozens of chips merge into a single unit via NVLink. That is precisely the actual selling point against the competition. When a report mentions figures like 'terabytes per second,' it is almost always about interconnect bandwidth.

The term also comes up in data center projects and standardization disputes. An alliance of several manufacturers is working under the name Ultra Ethernet to create open alternatives to proprietary connections. Behind this lies tangible self-interest: whoever controls the standard binds customers to itself in the long run.

In everyday life, this technology is encountered indirectly. The fact that a chatbot responds within seconds is also due to a model being distributed across multiple chips that are tightly coupled. For personal computers, this plays no role, since they contain a single processor. Interconnect is a matter of scale, and that scale has grown enormously in AI in recent times.