vLLM

A language model is a program that generates text word by word. To do this, it must run on powerful computing chips, usually graphics cards. This is exactly what vLLM is for: a freely usable piece of software that takes in the finished model and processes the requests of many users at the same time. It ensures that the expensive hardware is utilized as fully as possible. vLLM has been developed since 2023 at the University of California, Berkeley. Today it is one of the most widely used tools of its kind and is being further developed by a large community.

Why a piece of software decides the computing bill

Operating a language model costs money continuously. Every single answer consumes computing time on a graphics card, and such cards quickly cost 20,000 euros or more. Anyone who operates a model poorly leaves this hardware twiddling its thumbs half the time. Anyone who operates it well serves a multiple of users with the same card.

This is exactly where the value of vLLM lies. The software increases what is known as throughput, i.e. the number of words a card can output per second. In the developers' measurements, this figure was several times higher than that of older solutions. For a provider, this directly translates into lower costs per request.

On top of that comes a second point: independence. Anyone who uses a model via the interface of OpenAI or Google hands over their data and pays per use. With vLLM, a hospital or a bank can operate a freely available model on its own servers. The data never leaves the building. This is one of the reasons why vLLM appears particularly often in Europe.

PagedAttention: memory like a library shelf

While a model is writing a text, it has to remember what has been said so far. These notes reside in the graphics card’s memory and are called the KV cache. They are the actual bottleneck, because they grow with every word generated. Older systems reserved a large block in advance for each request, sized for the worst case. If the answer was short, the rest went unused.

vLLM solves this with a method called PagedAttention. Memory is broken up into many small compartments of equal size. Each request only gets as many compartments as it currently needs, and the compartments don’t have to lie next to each other. A management table keeps track of which compartment belongs to which request. It’s like a library shelf: the books of one row stand scattered about, yet the catalog still knows where everything is.

As a result, memory waste drops from often more than 60 percent to just a few percent. The freed-up space is used for additional simultaneous requests. A second trick is called continuous batching: as soon as one request is finished, the next one immediately takes its place, instead of an entire group waiting on the slowest one. Together, both explain the speed advantage.

Who uses vLLM and what stands alongside it

In everyday life, you never notice vLLM directly. It runs in the background of chatbots, coding assistants, and search functions that are built on open models like Llama, Mistral, or Qwen. Many AI features in enterprise software are also operated this way. Anyone who installs a model on a server themselves usually sets up vLLM with just a few commands.

What’s practical here is that vLLM offers the same interface as OpenAI. A program that previously addressed ChatGPT can often be redirected to one’s own model simply by changing the address. This significantly lowers the barrier to switching.

In the news, the name usually comes up in connection with costs or benchmarks. Competitors are SGLang, Nvidia's TensorRT-LLM, and Ollama, which is more geared toward private users. A common misconception: vLLM is not an AI model and does not train one either. It is purely operational technology for finished models, comparable to a car’s engine control unit, not the car itself.