local inference

When you use ChatGPT in the browser, the actual computational work happens on a server in a data center far away. Your request travels across the internet, a foreign computer evaluates it, and the result comes back. With local inference, it’s different: the AI model runs directly on your own device – the laptop, smartphone, or a special chip. No data leaves the device, and an internet connection is not necessary. This sounds like a technical detail, but it has far-reaching consequences.

Privacy and independence from providers

The most obvious advantage is privacy. Anyone who sends a request to a cloud service hands over that information – at least temporarily. For confidential texts, doctor’s notes, or business documents, this is often unacceptable. With local inference, no external provider sees what is being processed.

On top of that comes independence. Cloud services can raise their prices, change their terms of use, or simply go down. A model that runs locally still works even if the provider no longer exists or the internet connection is interrupted. This is especially relevant for companies that want to permanently integrate AI into their workflows without tying themselves to a single provider.

Why local inference was hardly possible for a long time – and now is

Powerful language models – that is, AI systems that understand and generate text – have long required amounts of computing power and memory that ordinary devices could not provide. A large model like GPT-4 takes up several hundred gigabytes in its raw state. That fits neither on an average laptop nor into the memory of a smartphone.

Two developments have changed that. First, more and more so-called small language models are emerging that, despite being significantly smaller, are surprisingly capable – Meta's Llama series or Microsoft's Phi models are well-known examples. Second, there are compression methods: in so-called quantization, the numerical values in the model are stored with less precision, similar to how a photo in lower resolution takes up less storage space. A model that would be 40 gigabytes uncompressed can thus be shrunk to under 5 gigabytes – with manageable losses in quality.

At the same time, chip manufacturers have adapted their hardware. Apple's M chips, Qualcomm's Snapdragon processors for smartphones, and Nvidia's consumer graphics cards now contain special processing units that handle AI computations particularly efficiently.

Local inference in products and news

Apple announced with “Apple Intelligence” that it would run many AI functions directly on the iPhone and Mac – explicitly as a privacy argument. Only if the task is too complex for the device is it forwarded to external servers, and Apple emphasizes that even these servers do not store the requests.

Outside of smartphones, there are programs like LM Studio or Ollama, with which technically interested users can install and try out freely available models on their own computer – without a cloud connection, without usage restrictions, without costs per request. In corporate environments, local inference is primarily used where sensitive data is processed: in law firms, clinics, or government agencies.

The term frequently appears in tech news in connection with the so-called “AI PC” – a device category that Microsoft and chip manufacturers have been actively marketing since 2024. This refers to computers with chips specially optimized for AI, meant to enable local inference quickly and with low power consumption. Whether this is a genuine turning point or primarily a marketing term is still being debated in the industry.