TensorRT

At its core, an AI model is a very long set of calculation instructions containing billions of numbers. This calculation usually runs on a graphics card, i.e. a specialized chip that carries out many computing steps simultaneously. TensorRT is a software package from the chip company Nvidia that reworks such a finished set of calculation instructions for its graphics cards. It rearranges the computing steps, merges them together, and selects the fastest known method for each step. The result delivers practically the same answers as before, but requires significantly less time and power. So TensorRT doesn’t help with training a model, but only afterward, during everyday use.

Why operators fight for every millisecond

Training a large model happens once and costs a lot of money. Using it afterward happens millions of times over. Experts call this usage step inference: question in, calculation, answer out. Anyone running a chatbot with many users pays computing time for every single answer. If an answer is computed a third faster, the bill drops accordingly.

On top of that comes user patience. A search query that hangs for three seconds feels broken. With an assistance system in a car or a camera that recognizes people, it’s even a matter of safety. Such systems must respond within fixed time windows, otherwise they’re useless.

A second reason is hardware scarcity. Powerful graphics cards are expensive and often hard to obtain. If an optimized model can handle twice as many requests per card, a provider needs only half as many cards. This is why optimization software like TensorRT is among the most economically important and most inconspicuous parts of the AI industry.

What happens when the model is reworked

TensorRT takes a trained model and produces from it a so-called engine package that is precisely tailored to a specific chip type. One important trick is called layer fusion. Many small computing steps are merged into one larger one. This saves the trips between the computing unit and memory, and these trips often cost more time than the computation itself.

The second major trick is quantization. Normally, a model stores its numbers very precisely, for instance with 32 bits per number. TensorRT instead computes with 16 or 8 bits, sometimes even coarser. You can imagine this like rounding prices to whole dollars: the calculation becomes faster, and the result stays almost the same. Only when the rounding is too coarse does quality start to suffer, which is why every model is checked after the rework.

In addition, TensorRT tries out several implementations for each computing operation and measures which one runs fastest on the available card. This measuring takes minutes to hours, but happens only once. The price for this is inflexibility: an engine package for one card generation often doesn’t run optimally, or at all, on another. And it works exclusively on Nvidia hardware, which ties providers to this manufacturer.

TensorRT in products and headlines

TensorRT is rarely visible because it works beneath the surface. It’s found in voice assistants, in translation services, in medical image analysis, and in driver assistance systems. Image generators on personal computers also use it to produce an image from a text prompt within a few seconds. For large language models there is a dedicated variant called TensorRT-LLM.

The name appears in business news when Nvidia’s market position is at issue. The company sells not only chips but also the software ecosystem that goes with them. This combination makes switching to competing chips costly, because a lot of optimization work would have to be redone. Analysts see this as a significant part of the lead Nvidia holds in the AI business.

A common misconception is that TensorRT is an AI model or a competitor to systems like ChatGPT. It is neither one nor the other. It’s better compared to a tool that tunes a finished engine for a particular racetrack. The engine stays the same, it just runs more smoothly.