On-Device Inference

When a computer program answers a question or recognizes a photo, the computation has to happen somewhere. Most of the time this doesn’t happen on your phone, but on powerful computers in large halls called data centers. Your device sends the request there over the internet and gets the result back. With on-device inference, this computation instead runs entirely on the device itself, meaning the phone, the laptop, or in the car. The word part inference refers to the use of a finished model: it no longer learns anything new, it only applies what it has previously learned. Nothing leaves the device, and it still works without an internet connection.

What speaks in favor of the device

The most important advantage is privacy. Your voice recordings, photos, or messages don’t need to leave anywhere if the analysis happens locally. For a keyboard that suggests your next word, this makes a noticeable difference. Otherwise every word you type would travel to a foreign server.

The second advantage is speed. A request to a data center and back often takes a hundred milliseconds or more, just from the network route alone. For a driver assistance system meant to detect a pedestrian, that’s too slow. Locally, this waiting time disappears, and the feature keeps working even in a dead zone or on a plane.

For manufacturers, money matters too. Every request to a data center costs electricity and server time, and that’s true for every single use. If the model runs on the customer’s device, the customer pays for the power from their own battery. Across hundreds of millions of devices, that’s a massive difference in operating costs.

How a large model fits on a phone

The fundamental problem is space. A powerful language model consists of many billions of stored numbers, called parameters. At full precision, this would require several hundred gigabytes of memory, while a phone might have around eight. So the model has to become smaller before it can even start.

The most important tool for this is called quantization. Here, each number is stored more coarsely, for example with four digits instead of sixteen in memory. This makes the model many times smaller and it loses surprisingly little quality. A second approach is distillation: a large model trains a small one to imitate its answers. The small model then might have three billion parameters instead of four hundred billion, but that’s enough for many everyday tasks.

On top of that comes specialized hardware. Modern phone chips contain a dedicated computing unit just for AI tasks, often called an NPU. It is tailored to exactly the kind of multiplications a model needs, while consuming little power. A common misconception is that local models are therefore equivalent to the large ones from the cloud. They are weaker at complex tasks, but immediately available.

From camera filters to translation on a plane

You probably use on-device inference every day without noticing it. Face recognition when unlocking, noise cancellation during video calls, and the automatic sorting of your photos by people all run locally. Live captions and offline translation belong here too. With voice assistants, it’s often split: detecting the wake word happens on the device, while the actual answer is then fetched by the phone from the network.

In business news, the term comes up in the context of chip makers and phone manufacturers. Apple, Google, Qualcomm, and Samsung advertise that their new chips can run larger models locally. Analysts are interested in whether this reduces the need for data centers or whether both grow side by side.

In practice, a mixed form usually prevails, often called a hybrid approach. The device answers simple requests itself, and passes difficult ones on to the server. Ideally the user notices nothing of this, except that some features are still there even offline.