TensorRT-LLM

Programs like ChatGPT are based on huge computational models that generate text word by word. These models don’t run on ordinary processors, but on graphics cards, because those can handle very many calculations at the same time. For a model to run truly fast on a particular graphics card, its computational steps have to be adapted to exactly that hardware. This is precisely what TensorRT-LLM is for, a freely available software from the company Nvidia. It takes in a fully trained language model and builds from it a version optimized for Nvidia graphics cards. Afterward the model can’t do anything more than before, it just answers faster and needs less power.

Why response speed decides over cost

A language model is trained once, but used millions of times. Every single answer costs computing time on expensive hardware. This usage phase is called inference. Anyone operating a chatbot with many users ends up spending significantly more money on inference over time than on training.

That’s why every speedup is directly worth money. If a graphics card can handle twice as many requests per second thanks to better software, a provider needs only half as many cards. At prices of several tens of thousands of euros per data-center graphics card, that’s an enormous difference. Nvidia advertises TensorRT-LLM with exactly these kinds of numbers.

For Nvidia, the software also serves a second purpose. It only works with Nvidia hardware. Anyone who builds their systems on it cannot easily switch to chips from AMD or other providers. This lock-in through software is considered an important reason for Nvidia’s strong market position.

From model to optimized computation engine

TensorRT-LLM works like a translator with a conversion assignment. You give it a trained model, such as Llama or Mistral. What comes out is a so-called engine, a file containing computational instructions precisely tailored for a particular graphics card. This conversion takes a few minutes and happens only once before operation.

During the conversion, several tricks work together. One is called quantization: numbers in the model are stored more coarsely, for instance with 8 instead of 16 digits of precision. This saves memory and computing time, but costs a bit of quality. Another trick combines many small computational steps into one large one, so the graphics card has to shuffle data back and forth less often.

Also important is how multiple user requests are served at the same time. TensorRT-LLM uses a method in which completed requests are immediately replaced by new ones, instead of waiting for the slowest one in the group. You can picture this like a supermarket checkout where the next person immediately steps up. This keeps the graphics card continuously utilized.

Where the software operates in the background

As an ordinary user, you never see TensorRT-LLM. It runs in data centers behind chatbots, translation services, and coding assistants. When an answer appears fluidly word by word instead of stuttering sluggishly, such optimization software is often behind it.

In business news, the name usually comes up in connection with Nvidia’s financial figures. Analysts then emphasize that Nvidia doesn’t just sell chips, but an entire software ecosystem around them. Competitors like AMD must not only build faster hardware, but also deliver comparable software. So far they have only partially succeeded.

A common confusion involves similar tools. vLLM is an open alternative that also runs on other hardware. Ollama is aimed at private individuals who want to try out a model on their own computer. TensorRT-LLM, by contrast, targets professional continuous operation, where every percent of speed counts.