Load Balancer

Large internet services don’t run on a single computer. Behind a website there are often dozens or hundreds of machines, all capable of handling the same task. A load balancer sits in front of this group of machines and is the first to receive all the users' requests. It then forwards each request to one of the machines that currently has little to do. For the user, this is invisible: they type in an address and get a response. The English name literally means “load distributor,” and that’s exactly what describes the task.

Why a single server isn’t enough

Every computer has a limit. It can only handle a certain number of requests per second. If this limit is exceeded, responses become slow or fail entirely. An online shop that crashes on Black Friday loses a lot of money within minutes. That’s why multiple machines are set up and the work is distributed among them.

The second reason is fault tolerance. Hardware breaks, software needs updating. If only one machine provides the service, the service is gone as soon as that machine fails. A load balancer, on the other hand, regularly checks whether each machine is still responding. If one stops responding, the distributor removes it from the list and no longer sends anything to it. Users notice nothing of the outage.

This is precisely why load distribution is also a cost question. Instead of a single, very expensive mainframe, you use many affordable standard machines. If demand grows, you simply add more. This way of growing is called horizontal scaling in tech, and it doesn’t work without a load balancer.

How the distributor makes its decision

The simplest rule is called round robin: the distributor goes through the machines in order, first request to machine one, second to machine two, and so on. This is fair, but rather blind. A request can be harmless or very computationally intensive. That’s why there are smarter rules, such as distributing to the machine with the fewest currently open connections.

Some machines are more powerful than others. For this, each machine is given a weighting, i.e., a share of the total load. A machine that’s twice as fast then receives twice as many requests. In addition, health checks run in the background: the distributor calls a test address on each machine at short intervals. Only those that respond correctly stay in the game.

A typical stumbling block is session data. If you log in and the machine only remembers this locally, the next request must not end up on a different machine. Otherwise you’d suddenly be logged out. This is solved either by permanently assigning a user to one machine or, better, by having the machines store their session data in a shared database. The load balancer should not be confused with a CDN: a CDN caches content in many locations around the world, while the distributor merely splits work across machines.

From movie night to AI chatbot

Practically every major platform sits behind a load balancer. When millions of people start a series simultaneously in the evening, such systems distribute the traffic. Whether a site holds up during concert ticket sales or on a train app during a strike day also depends on this technology. Cloud providers like Amazon, Microsoft, or Google sell load balancing as a ready-made service that can be switched on with just a few clicks.

In the AI world, you encounter this term particularly often. A chatbot answers requests on expensive graphics cards, and every answer costs computing time. A distributor decides which graphics card handles the next request. Reports about overloaded AI services or waiting lists for new models often revolve around exactly this point.

For investors, this is relevant because load distribution is a cost lever. Well-distributed requests mean less unused hardware and thus higher profit margins in the cloud business. The term rarely appears literally in quarterly reports, but it lies behind phrases like improved data center utilization.