Batch Processing

A computer can accept tasks in two ways. Either it handles every request immediately as soon as it arrives. Or it collects many requests and works through them together later. The second variant is called batch processing. A batch is simply a stack of similar tasks that are processed together. The counterpart is called real-time processing: there, every second counts, while with batch processing, overall throughput counts.

Why batches are cheaper than single pieces

Almost every machine works more efficiently when it does one thing repeatedly. A bakery bakes a hundred rolls in one oven run, not in a hundred runs. Heating up the oven costs the same regardless of how full it is. It’s similar with computers: preparation, data loading, and switching cost time, regardless of the quantity.

This becomes especially clear with graphics cards, the specialized chips behind almost every modern AI. They contain thousands of small computing units that can work simultaneously. If you send them a single task, most of the chip remains unused. If you send them thirty tasks at once, they run at nearly full capacity. Computing time barely increases, but throughput rises massively.

That’s why batch processing is above all a matter of cost. Providers of AI services often charge only half the normal price for batch jobs. The deal is simple: you give up an immediate answer, and in exchange it becomes cheaper. Anyone wanting to have a million product descriptions translated saves a great deal of money this way.

From queue collection to the finished batch

Technically, a queue is needed first. Incoming tasks are placed there instead of being processed immediately. A control program then decides when a batch is full enough. Typical rules are: start as soon as 64 tasks have accumulated, or after five seconds at the latest. Then the entire batch is sent together to the computer.

The size of the batch is called batch size. It is the central control knob of this process. A large batch means low cost per task, but a longer wait time for the individual. A small batch responds quickly but wastes computing power. In addition, a large batch requires more memory, because all the data must be present at the same time.

With chatbots there is a particular difficulty. Responses vary in length, so one finishes earlier than another. Modern systems therefore use continuous batching: as soon as a slot in the batch becomes free, a new request moves in. The batch stays permanently full, instead of waiting for the slowest participant.

From bank accounts to nightly model evaluation

Banks are the classic example. Transfers are collected during the day and posted in large runs at night. That’s why a payment sometimes doesn’t appear in the account until the next business day. Payroll statements, electricity bills, and phone bills are also produced in such nightly runs.

In the AI world, you encounter the term in two places. During training, a model doesn’t learn from one example after another, but always from a batch at once. When operating the finished model, providers bundle the requests of many users into a batch. You don’t notice this, but it significantly lowers the costs behind your chat window.

A common misconception is that batch processing is old-fashioned and real-time is always better. In fact, the two complement each other. An emergency call system or a brake assistant must never collect and wait. Evaluating ten million customer comments, on the other hand, doesn’t need to be done in one second. The right question isn’t what’s faster, but how fast it really needs to be.