Supercomputer Cluster

A single computer has a fixed limit: at some point it becomes too slow for the task at hand. The solution is to connect many computers together so they can work jointly on the same task. That is exactly what a supercomputer cluster is: hundreds to hundreds of thousands of individual machines standing in racks in a hall, connected by extremely fast cables. From the outside they act like a single giant machine. The individual machines within are called nodes. Anyone who speaks of a supercomputer today almost always means such a cluster and no longer a single large machine.

Why AI companies fill entire halls

Training a large language model is a computing task that a normal computer would not manage in thousands of years. If you distribute the same work across 20,000 chips, it shrinks to weeks. That is precisely why access to compute clusters has become a fierce competitive factor. Companies like OpenAI, Google, or Meta build or lease facilities worth billions.

This also explains why compute clusters keep showing up in business news. Whoever does not have such a facility simply cannot build certain models, no matter how good their ideas are. The chipmaker Nvidia has become one of the most valuable companies in the world partly for this reason. States are also investing because they do not want to be completely dependent on American providers.

A second reason is electricity. Large clusters consume as much energy as a small city, sometimes over 100 megawatts. That is why they are built where electricity is cheap and the grid can bear the load. In some regions, power connections have become scarcer than the chips themselves.

From node to cluster

A cluster is built up in layers. At the bottom is the node, a server with processors and usually several graphics chips. Several nodes stand in a rack, and many racks form rows in a hall. Everything is connected by a network that is significantly faster than any home network.

This network is actually the difficult part. During training, the chips must constantly reconcile their intermediate results with one another. If the connection is too slow, expensive chips wait for data and sit idle. This is often compared to group work: if everyone is only coordinating results, nobody gets to actually work. That is why the cabling of a cluster often accounts for a significant portion of the total price.

Software called a scheduler distributes the jobs across the available nodes. It ensures that multiple users can share the facility. In addition, a cluster must cope with failures: with tens of thousands of components, something breaks every day. Training runs therefore save intermediate states regularly so that one does not have to start over from scratch after a defect.

Where these facilities are located and who uses them

Most people use clusters every day without noticing. Every request to a chatbot ends up in a data center and is answered there by powerful chips. Weather forecasts, computer crash tests, and the search for new medications also run on such facilities.

In Germany, the Hawk and the Hunter in Stuttgart, as well as facilities in Munich and Jülich, are among the best known. The European computer Jupiter in Jülich reaches the mark of one trillion computing operations per second. Universities are allocated computing time upon application, similar to grant funding.

A common misconception: a cluster does not make every program faster. Only tasks that can be broken down into many parallel parts benefit from it. A video game does not run better on a supercomputer than on a good home PC. That is why the figure for raw computing power alone says relatively little.