Inferring

A computer program that learns from examples goes through two very different phases. First, it learns: it is fed enormous amounts of text or images and adjusts itself until it delivers suitable answers. This learning phase can take weeks and happens only once. After that, the second phase begins, and it is called inferring. Inferring means: the finished program receives a concrete input, computes it through once, and outputs a result. The word comes from the Latin infere, meaning roughly “to conclude” — the program draws a conclusion about the new case from what it has learned.

Why every single answer costs money

During learning, a company pays a very large sum once. During inferring, on the other hand, it pays a little bit again and again. Every request from a user means computing time on expensive specialized chips, running somewhere in a data center. For a chatbot with hundreds of millions of users, these small amounts add up to enormous sums.

That’s why inferring is one of the most important cost factors in the AI industry. The term appears regularly in business reports and news when profit margins are discussed. A provider can have a brilliant model and still make a loss, because every answer costs more than the customer pays. A large part of current research therefore does not focus on making models smarter, but on making them cheaper to run.

Besides money, time plays a role. No one likes waiting ten seconds for a search engine’s answer. The delay between input and first output is called latency, and it arises almost entirely during inferring. Fast inference is therefore also a quality feature, not just a cost-saving measure.

What actually happens during the computation

A model consists of many billions of stored numbers, the so-called parameters. They are the result of the learning phase and no longer change during inferring. The input is converted into numbers and then computed layer by layer through this network of parameters. In the end, there is a probability for which word or which answer comes next.

For text models, this happens word by word. The model generates a piece of text, appends it to the input, and computes again. That’s why in chat programs you can see the answer slowly emerging. For a paragraph of a hundred words, the model therefore runs through roughly a hundred times.

There are several tricks to speed this up. With quantization, the parameters are stored more coarsely, for example with eight instead of sixteen decimal places — the model becomes minimally less accurate, but noticeably faster. With a design called Mixture of Experts, the model activates only a small part of its building blocks per request instead of all of them. Incidentally, a common misconception is that the model learns further while being used. It does not: if someone corrects an answer, nothing changes about the parameters.

From the phone to the data center

You encounter inferring every day, usually without it being called that. Face recognition when unlocking your phone is inference. So is automatic translation in the browser, voice control in the car, and the spam filter in your inbox. In all these cases, the model was trained somewhere beforehand and now merely runs.

An important distinction is location. Some models infer directly on the device, others send the request to a data center. On the device, the data stays private and it works offline, but only small models fit there. In the data center, large models are available, but this requires internet access and trust in the provider.

In business news, the term comes up mainly with chip manufacturers. There are processors specifically built for training, and others optimized for cheap inference. Analysts assume that the market for inference hardware will ultimately exceed that for training hardware. The reason is simple: training happens rarely, usage happens constantly.