Locally runnable model

When you type a question into ChatGPT, the actual computation doesn’t happen on your device. Your text is sent over the internet to large computers belonging to the provider company, processed there, and the answer comes back. A locally runnable model does exactly the opposite: the program sits as a file on your laptop or phone and computes there. It doesn’t need an internet connection to do this and sends your inputs to no one. The price for this is performance, since your device is far weaker than a hall full of specialized chips. That’s why local models are usually smaller, stripped-down variants of the well-known systems.

Privacy and independence from the provider

The strongest argument for local models is confidential data. A law firm may not simply upload client files to someone else’s servers. A doctor may not do that with patient data, nor a company with unpublished construction plans. If the model runs in-house, this problem largely disappears, because the data never leaves the device or the company network.

The second reason is independence. Anyone who builds their software on a cloud service – that is, on rented computing power on the internet – depends on that service’s prices and rules. The provider can raise prices, shut down a model, or roll out a new model with different behavior. A locally stored model doesn’t change on its own. It will still work exactly the same way in five years.

On top of that come practical advantages. There are no costs per request, only a one-time cost for the hardware. And it works where there’s no network: on a plane, in a mine, in a car without reception. That’s why car manufacturers and phone makers build small models directly into their devices.

How a model fits on a laptop

A language model consists of billions of stored numbers, the so-called parameters. They are what the model retained from learning from texts. When generating a response, all these numbers must fit into fast working memory, otherwise it becomes extremely slow. This determines what can even run on an ordinary device.

Two tricks make this possible. First, you start with smaller models from the outset: not several hundred billion parameters, but three, seven, or twenty billion. Second, you store each individual number more coarsely, often with four instead of sixteen binary digits. This process is called quantization and shrinks the file to roughly a quarter of its size. The model becomes somewhat less precise in the process, but remains usable for most tasks.

A rough sense of scale: a model with seven billion parameters takes up about four gigabytes when quantized and runs smoothly on a modern laptop. The fact that you’re even allowed to download it at all is due to open weights. This is what models are called whose parameter files the developers release publicly, such as Llama from Meta, Mistral from France, or Qwen from Alibaba. With ChatGPT or Claude, this is not the case – nobody gets those files.

From phone assistant to server in the basement

You encounter local AI most often without it being labeled as such. The automatic translation in airplane mode, the dictation function, and the text suggestions on your phone keyboard often run directly on the device. Apple calls its variant Apple Intelligence and only forwards difficult requests externally. Such mixed solutions are called hybrid systems.

Anyone who wants to try it out themselves uses programs like Ollama or LM Studio. You download a model file and chat in a window that looks like a normal chat service. It’s just that no server is involved. In companies, the same principle runs on an in-house server that summarizes contracts or completes program code.

In the news, local models mainly come up as an economic topic. Every request that stays on the device is a request for which no cloud provider gets paid. A common misconception, by the way, is that local automatically means worse. For summaries, translations, or simple programming help, small models are often entirely sufficient. But for long chains of reasoning and difficult specialized questions, the large systems from the data center remain clearly superior.