On-Device AI

When your phone enhances a photo or turns your speech into text, that computation has to happen somewhere. Often this doesn’t happen on the device itself, but in a large data center: a hall full of computers reachable over the internet. With On-Device AI, the computation instead runs on the device you’re holding in your hand. Your data never leaves the phone. The downside: a phone has far less computing power than such a hall. So the programs have to be built much more efficiently.

Why manufacturers move the computation onto the device

The first reason is privacy. Anyone who sends their voice messages or photos to a remote server has to trust the operator. If everything stays on the device, this trust problem doesn’t arise at all. Apple, Google, and Samsung therefore explicitly advertise their AI features with this argument.

The second reason is money. Every request to a data center costs the provider electricity and computing time. With a billion users, this adds up to enormous amounts. If the device computes it itself, the user pays for the electricity out of their own battery. For manufacturers, this is a business model that doesn’t get more expensive as the number of users grows.

The third reason is reliability. In a dead zone, on an airplane, or in a tunnel, a cloud feature simply doesn’t work. A translation that runs on the device works everywhere. It also responds faster, because the trip to the server and back is eliminated.

How a large model fits into a phone

The largest language models have hundreds of billions of so-called parameters. These are the adjustable numbers in which a model stores its knowledge. Something like that doesn’t fit into a phone. Models designed for devices therefore typically have one to eight billion parameters. They are weaker, but completely sufficient for many tasks.

In addition, the numbers are stored more coarsely. Instead of using 16 bits per parameter, one computes with 8 or 4 bits. This process is called quantization. You can think of it like an image with fewer colors: it still looks good, but takes up much less space. A model can shrink this way from 16 to 4 gigabytes.

Modern phone chips also contain a dedicated component just for such computations, often called an NPU. It’s specialized for the simple multiplications that make up AI models, and consumes little power in doing so. Many systems work in a hybrid way: simple requests are handled by the device, complicated ones go to the cloud. The user usually doesn’t notice this division at all.

On-Device AI in phones, cars, and quarterly earnings

The best-known examples are found in smartphones. The live translation of a phone call, removing people from photos, dictation, and the keyboard’s text suggestions run largely locally on current devices. Facial recognition for unlocking also belongs here, and has done so for years.

Outside the phone, you encounter this principle in cars, where assistance systems have to recognize traffic signs — a one-second wait for a server would be dangerous here. Robot vacuum cleaners, cameras, and hearing aids also use small models on the device. In this context, you often come across the related term edge computing, which generally refers to computing at the edge of the network, close to the user.

In business news, On-Device AI is an important argument for chip manufacturers like Qualcomm, Arm, or Apple. They hope customers will buy new devices just to be able to use the AI features. A common misconception, by the way, is that On-Device AI keeps learning on the device. The model is usually delivered fully trained and only performs computations from then on.