Continuous Batching
Continuous batching is a technique that allows a server to process many requests to an AI language model at the same time, immediately replacing completed requests with new ones. This keeps the expensive graphics chips idle less often, and users wait less time for their answers.
When you ask a chatbot a question, it lands on a server with specialized computing chips. These chips work fastest when they process many requests at once instead of one after another. Such a group of simultaneously processed requests is called a batch. In the classic approach, the server waits until all requests in the batch are finished before starting the next one. Continuous batching breaks with this rigid rule: as soon as a single request is completed, a waiting one immediately takes its place. The batch is thus continuously refilled instead of being completely swapped out.
Why graphics chips otherwise sit idle
The computing chips in AI data centers are extremely expensive. A single accelerator can easily cost as much as a mid-range car. Every second it isn’t computing is money lost. This is exactly the problem that arises with the rigid batch procedure.
The reason lies in the very different response lengths. One request might simply be: What time is it in Tokyo? The answer is finished after ten words. Another request demands a two-page summary of a text. If both are in the same batch, the short request blocks a slot even though it has long since been completed. The chip then keeps computing for an empty slot.
In practice, this can halve throughput or worse. Operators measure it in tokens per second, meaning text chunks per second. Continuous batching increases this value manyfold depending on utilization. For users this means shorter wait times, for the operator lower costs per answer. That’s why the technique is now standard equipment on every serious AI server.
Moving up in the batch
A language model doesn’t generate its answer all at once. It produces one piece of text after another, each in a separate computing step. The server can therefore decide anew after every single step who sits in the batch. This is exactly where continuous batching comes into play.
A management program, the scheduler, maintains a queue of all open requests. After each computing step, it checks which requests are finished and removes them from the batch. It immediately fills the freed-up slots with the next ones waiting. A fitting image is a chairlift: whoever gets off at the top frees up their seat, and the next person immediately gets on at the bottom. The lift never stops for this.
For this to work, the server must keep intermediate results in memory for every running request. This intermediate storage is called the KV cache and limits how many requests can run simultaneously. If memory isn’t sufficient, the scheduler must pause a request and resume it later. Continuous batching therefore solves the time problem, not the memory problem. It should also not be confused with simple dynamic batching, which only adjusts the batch size before it starts.
Where the technology is used
You will never see continuous batching directly, since it runs deep in the server room. It becomes noticeable as response speed. When a chatbot still writes smoothly word by word during peak times, such a technique is usually behind it. Without it, responses would noticeably stutter under high load.
In news and technical texts, the term appears mainly in connection with software for running language models. Well-known examples are vLLM, Nvidia's TensorRT-LLM, and Hugging Face's Text Generation Inference. These programs explicitly advertise continuous batching as a key feature. Nvidia calls its own variant in-flight batching, referring to the same principle.
The topic also plays a role in AI providers' pricing. Anyone using an interface pays per text chunk, and the price is tied to operating costs. Better chip utilization drives these costs down. Some of the price drops in AI services in recent years are due to such optimizations, not new models.
