Inference System

An AI model is initially just a very large file of numbers. On its own, this file can’t do anything. For it to become a usable service, software is needed that loads the model into memory, receives requests, and sends back answers. This exact software, together with the machines it runs on, is called an inference system. The English term “inference” here means: applying the finished model, as opposed to training it. You can think of the model as the engine and the inference system as the rest of the car – transmission, fuel tank, steering.

Why this is where cents per answer get decided

Training a large model happens once and costs a lot of money. Operation, by contrast, runs around the clock. For a chatbot with millions of users, tiny amounts per answer quickly add up to millions per month. A well-built inference system can utilize the same hardware two or more times as efficiently as a poorly built one. That’s why it’s often more important to providers than the last bit of model quality.

For users, waiting time matters most. Two figures are measured separately here. One is the time until the first visible word appears, the other is the speed at which the text continues afterward. Both depend almost entirely on the inference system, not on the model itself. A service can feel sluggish even though the model is excellent.

Then there’s reliability. Graphics cards fail, requests arrive in waves, and during peak times everyone wants to know something at once. An inference system has to absorb these fluctuations without losing requests. This is exactly why large providers build dedicated teams for it.

From request to finished text

First, a request lands in a queue. The system collects several requests and sends them through the graphics card together. This bundling is called batching and is the single most important trick there is. A graphics card, after all, computes barely any slower for twenty requests than for a single one. Modern systems even keep rebatching continuously, as soon as one request in the batch finishes.

A second building block is the cache for already-processed text. A language model generates words one after another and would theoretically have to recompute the entire text so far each time. Instead, the system remembers the intermediate results. This cache is often managed with great effort, because it can quickly take up several gigabytes. If the model doesn’t fit on a single graphics card, the system also distributes it across multiple cards.

The inference system is often confused with the model itself. You can run the same model on different systems – with markedly different speed and price. Conversely, a whole range of different models often run side by side within one system. Both are separate building blocks.

Where these systems work in everyday life

Every time you use an AI chat, request an automatic translation, or your phone describes a photo, an inference system is working in the background. On phones, it sometimes runs directly on the device; for large models, it almost always runs in a data center. Search engines, spam filters, and recommendations in streaming services depend on it too.

In the news, this topic usually comes up indirectly. When a provider lowers prices per request or Nvidia talks about demand for compute chips, it’s essentially about inference. Well-known building blocks of such systems are called vLLM, TensorRT-LLM, or Triton. Cloud providers sell access to these as ready-made services, so companies don’t have to build anything themselves.