Load Balancing

Large internet services never run on a single computer. Behind a website there are often hundreds or thousands of machines, all running the same program. Load balancing is the technique that distributes incoming requests across these machines. A dedicated device or program sits in front of the group, accepts every request, and forwards it to one of the machines. The goal is for all of them to receive roughly equal amounts of work so that none collapses under the load. The English term literally means load equalization, and that is exactly what it is about.

Why a website collapses without a distributor

A single server can only answer a limited number of requests per second. If more come in, queues build up. Response times rise, and eventually the connection drops. For an online shop this means direct revenue loss; for a bank it means failed transfers.

The second reason is fault tolerance. Hardware breaks, software crashes, updates require restarts. A load balancer regularly checks whether each machine is still responding. If one stops responding, the balancer simply removes it from the list. Users notice nothing, because their requests go to the remaining machines.

Third, load balancing allows for flexible growth. Instead of buying an ever more expensive mainframe, you add more ordinary servers. This strategy is called horizontal scaling. It is the reason why modern data centers consist of many similar machines rather than a few huge ones.

The rules by which distribution happens

The simplest rule is called round robin: the distributor goes through the servers in order, one request for each, then starts over from the beginning. This is uncomplicated, but it ignores the fact that some requests are much more demanding than others. Clicking on an image costs less computing time than a complex database search.

Smarter methods therefore look at the state of the servers. With “least connections,” the request goes to the machine that currently has the fewest open connections. Other methods measure response time or processor load. Servers can also be given weights: a new, faster machine then receives twice as many requests as an older one.

A typical problem is session data. If you log in and your shopping cart is stored only on server 7, the next request must not end up on server 12. One solution is session stickiness: the distributor remembers where a user belongs and always sends them there. The cleaner solution is to store such data in a shared storage that all servers can access. Then it doesn’t matter who handles the request.

From streaming services to AI data centers

Every major service you use works with load balancing. Streaming platforms distribute video requests to servers near you. Online games push players onto different instances. When ticket sales for a concert start and the site holds up despite millions of visits, good load balancing is behind it. If it collapses, you read about it in the news the next day.

In the AI industry, the term is doubly present. On one hand, providers like OpenAI or Google distribute requests to their chatbots across huge numbers of graphics cards. On the other hand, load balancing also appears within the models themselves: in a mixture-of-experts model, a small auxiliary network decides which part of the model handles a given request. If it distributes poorly, some parts become overloaded while others sit idle. Developers therefore build in a so-called load-balancing loss, which penalizes uneven distribution during training.

At cloud providers, load balancing is nowadays a product you can set up with just a few clicks. There it’s called something like an Elastic Load Balancer and is often combined with autoscaling: as load increases, new servers start automatically, and the balancer immediately incorporates them. A common misconception is that load balancing is the same as a content delivery network. A CDN caches content close to the user, while a load balancer decides which machine takes on the work.