Inference Time

A program that writes texts or recognizes images first learns its capabilities from huge amounts of data. This learning phase is called training and is eventually completed. After that, the finished program is used: you provide an input, it computes, and a result comes out. This exact usage is called inference. Inference time is the duration a single such computation takes — from input to finished output. It is usually measured in milliseconds or seconds.

Why milliseconds determine cost and usability

Inference time is the difference between a tool people enjoy using and one that annoys them. With a search engine, users expect an answer in under a second. With a chatbot, people tolerate a bit more, as long as the text immediately starts flowing. In a self-driving car, recognizing a pedestrian must not take longer than a few milliseconds. Here, a slow response is not an inconvenience but a safety problem.

Then there is the economic side. Every second of computing time occupies an expensive specialized chip in a data center. Training a large model is a one-time investment, whereas inference runs continuously. A provider with millions of users pays anew for every single response. If it halves the inference time, the same hardware can serve twice as many requests.

A common misconception: a better model is automatically a slower one. That’s not necessarily true. A larger model does need more computation steps, but good architecture and suitable hardware can offset the difference. Conversely, a small, poorly optimized model can be surprisingly sluggish.

What makes the computation faster

For language models, inference time breaks down into two parts. First, the model reads in the input, which happens in one go. Then it generates the response word by word, or more precisely in small text chunks called tokens. Each new token requires its own computation pass. That’s why a distinction is made between the time until the first visible character and the speed at which the text continues afterward.

There are several levers to save time. Quantization stores the model’s internal numerical values more coarsely, for example with eight instead of sixteen digits of precision. This makes the model compute faster while losing only a little quality. Another approach is distillation: a large model teaches a small one to behave similarly. The small model then responds noticeably faster.

A third lever is bundling, known in technical jargon as batching. Instead of processing each request individually, the server computes many at once. This uses the chip more efficiently and lowers the cost per request. For the individual user, however, the waiting time can slightly increase. Providers therefore have to weigh throughput against response time.

Inference time in products and headlines

One encounters this topic constantly without the term ever being mentioned. When a chatbot builds its answer character by character, that’s not a stylistic effect but the visible inference time. When the translation app works more slowly offline than online, it’s due to the phone’s weaker computing power. Even a phone keyboard’s autocomplete must deliver its suggestions within a few milliseconds.

In the news, inference time usually appears in disguise. Companies advertise with tokens per second or with low latency, meaning short delay. Chip manufacturers compare their accelerators precisely on these figures. And when a provider lowers the prices for using its models, a reduced inference time is almost always behind it.

It’s important to distinguish this from training time. Training time occurs once and is measured in weeks and millions of dollars. Inference time occurs with every use and is measured in milliseconds. Over the lifetime of a successful product, inference often adds up to the larger cost block.