Scheduling

A computer almost always needs to do more things at once than it can truly do simultaneously. The browser wants to load images, the operating system checks for updates, a program calculates in the background. Because the computing components are limited, someone has to determine the order. That is exactly what scheduling is: the decision about which task goes next, how long it may run, and on which component it runs. A small management program, the scheduler, is in charge of this. You can imagine it like the person at the reception of a doctor’s office: they decide who waits, who is seen immediately, and who goes into which treatment room.

Why the order determines waiting time and costs

Computing time is expensive, especially on graphics cards for AI. A modern AI graphics card costs several tens of thousands of euros. If it sits idle because no task has been assigned to it, the operator is burning money. Good scheduling keeps the expensive components busy as continuously as possible. Large providers regularly report utilization jumps of ten or twenty percent achieved solely through better allocation.

For users, what matters most is the perceptible waiting time. When you ask a chatbot something, your request lands in a queue with thousands of others. The scheduler decides whether your answer begins after one second or after ten. That’s why scheduling is not a purely technical detail, but something directly experienced.

There is a trade-off involved. You can either maximize utilization of the machines or guarantee short waiting times for all users, but rarely both at once. Anyone who wants full utilization collects tasks and processes them in batches. This increases waiting time for individuals. Every scheduler is therefore a compromise that operators deliberately configure.

Queues, priorities, and time slices

The simplest rule is the line at a kiosk: first come, first served. That’s fair, but unwise if one huge order blocks all the small ones. That’s why real schedulers work with priorities. Urgent tasks get a higher rank and are allowed to jump ahead. A keyboard input is more important than a backup, which can also finish overnight.

A second basic mechanism is time slices. Each task only gets a few milliseconds of computing time, then the scheduler interrupts it and takes up the next one. This constant switching is called preemption. Because it happens very quickly, it appears to us as true simultaneity, even though in reality calculations happen one after another.

In AI systems, there’s an additional special feature: batching. Instead of processing individual requests one after another, the scheduler bundles several together. The graphics card then computes for many users in one pass, which is significantly more efficient. A common misconception, by the way, is equating scheduling with load balancing. Load balancing distributes load across multiple machines, while scheduling governs the order within a single machine. In practice, the two work together.

From smartphones to AI data centers

Scheduling is embedded in every device you use. Your phone keeps the video running smoothly while messages sync in the background. The fact that a battery lasts longer is often due to the scheduler batching tasks and letting the processor sleep in between. Even the claim that a game runs more smoothly on a new chip often has to do with better allocation.

In business news, the term usually comes up in connection with data centers. Software like Kubernetes or Slurm distributes thousands of tasks across entire server fleets there. When a cloud provider announces it has increased the efficiency of its graphics cards, it’s often new scheduling behind it, not new hardware. This is economically attractive because software is cheaper than buying additional chips.

For you as a user, scheduling remains invisible as long as it works well. It only becomes noticeable when there are disruptions: hanging programs, sudden stutters, or a chatbot that doesn’t respond for minutes under load. Such moments show that the distribution of computing time isn’t working out at the moment.