Inference Engine

At its core, an AI model is a huge collection of numbers and computational rules. For an answer to a question to emerge from this, someone has to actually carry out these calculations. That’s exactly what an inference engine does: a program that runs a fully trained model and answers user requests. It doesn’t teach the model anything new, it merely uses it. You can think of it as an engine control unit: the engine is built, and the control unit ensures it runs smoothly and efficiently. Well-known examples of such programs are called vLLM, TensorRT-LLM, or llama.cpp.

Why running a model is more expensive than training it

Training a large model, that is, having it learn from example data, costs a lot of money once. Running it, on the other hand, costs money every single day, for every single request. A chatbot with millions of users answers billions of questions daily. Each one occupies an expensive graphics card for a fraction of a second. Over the course of a year, these costs often clearly exceed the training costs.

That’s why the inference engine is an economic lever. A good engine gets two to four times as many answers out of the same hardware as a poor one. The model itself remains unchanged; only the surrounding software is better. For providers, this directly translates into: lower prices per request or higher profit margins.

A second point is waiting time. Users abandon a request if an answer only appears after ten seconds. That’s why engines optimize not just for overall throughput, but also for how quickly the first word appears on screen.

The tricks under the hood: batching, caching, and coarser numbers

The most important trick is called batching. The engine collects several requests from different users and sends them through the graphics card together. A graphics card computes many tasks in parallel almost as fast as a single one. Modern engines can even insert new requests while others are still running. This is called continuous batching.

The second trick is a temporary storage area, the KV cache. A language model writes word by word and in doing so must repeatedly take the preceding text into account. Without storage, it would have to recalculate these intermediate results every single time. Instead, the engine keeps them on hand. Because this storage quickly becomes enormous, good engines manage it in small blocks, similar to how an operating system manages working memory.

Third, the engine often stores the model’s numbers more coarsely, for example with 8 instead of 16 decimal places of precision. This process is called quantization. This makes the model smaller and faster, but it loses a bit of precision. A common misconception is that the engine makes the model smarter. It doesn’t change the model’s capabilities, it only changes speed, cost, and sometimes minimally the quality of the answers.

From the cloud to your phone

Every time you use ChatGPT, Gemini, or a translation app, an inference engine is running in the background. It’s invisible, but it determines how quickly the answer appears. Image search on your smartphone or voice recognition in your car work the same way.

On small devices, there are dedicated engines. Programs like llama.cpp or Ollama let a language model run on an ordinary laptop, entirely without internet access. The advantage: the data never leaves the device. The disadvantage: only smaller models fit into memory.

In business news, the term comes up in the context of the market surrounding AI. Companies like Nvidia sell not just chips, but also the matching software. Start-ups advertise that they can serve requests more cheaply than the competition. Anyone reading such reports should pay attention to two metrics: tokens per second, meaning output speed, and cost per million tokens.