Serving Stack

A fully trained AI model is at first just a very large file full of numbers. For millions of people to use it simultaneously, it needs an entire layer of software wrapped around it. This layer receives requests, loads the model onto the appropriate compute chips, distributes the workload, and sends the answer back. This exact bundle of programs is called a serving stack. The English term “to serve” means here “to deliver”: the model is delivered to users, just as a website is delivered to visitors. A serving stack is therefore not a single program, but a stack of building blocks layered on top of one another.

Why the stack determines cost and latency

Training a large model happens once and costs a lot of money in one go. Operation, on the other hand, continues every day. Every user request consumes electricity and occupies expensive specialized chips for a fraction of a second. If a provider answers hundreds of millions of requests daily, fractions of a cent add up to millions in cost.

A good serving stack can utilize the same hardware significantly better than a poor one. In practice, the difference between a naive solution and an optimized system is often a factor of five or ten in the number of requests per chip. The model itself remains completely unchanged in the process. That’s why the stack is an economic lever much discussed in the industry.

Then there’s latency. Users abandon a request if an answer doesn’t start appearing within ten seconds. So the stack must be both economical and fast at the same time, and these two goals often work against each other. Bundling very many requests into batches saves computing time, but makes individual users wait longer.

The building blocks between request and answer

At the very front there is usually an access point to which programs send their requests. Behind it sits a load balancer, which decides which server handles the request. Only after that comes the actual inference engine, the program that runs the model’s computations. Well-known examples of such engines are vLLM, TensorRT-LLM, and SGLang.

The most important technique inside is called batching: the stack collects multiple user requests and runs them through the model together. This pays off because, when processing a single request, the chips spend most of their time waiting for data. A comparison helps: a bus that carries ten people at once is cheaper per person than ten individual trips. Modern stacks continuously replace completed requests with new ones, instead of waiting for a full batch.

A second core building block is the cache for already-computed text segments, the KV cache. It prevents the model from having to recompute the entire preceding text with every new word. The stack must manage this memory cleverly, because it quickly fills up the scarce graphics memory. On top of that come monitoring, queues, and rules that throttle individual users if they send too many requests.

Serving stacks in products and headlines

Every time you use a chatbot or an app offers an AI feature, a serving stack is working in the background. Translations in browsers, image generators, and search suggestions also run through such systems. The stack usually only becomes visible during outages, for example when a service is overloaded and starts rejecting requests.

In business news, the term comes up when providers lower their prices per request. Such price cuts often aren’t due to new models, but to a better stack. Even companies that don’t train any model themselves make money by deploying other people’s models especially efficiently.

A common misconception is that the serving stack is simply the hardware. What’s actually meant, however, is the software layered on top of it; the hardware is merely the foundation. A useful distinction: inference is the computation itself, the serving stack is the organization surrounding it. Without it, even the best model would remain a file that no one can use.