Self-hosted model

Anyone working with an AI program like a chatbot today usually sends their inputs over the internet to a remote computer. This computer belongs to a provider, calculates the answer there, and sends it back. With a self-hosted model, exactly that does not happen. Instead, the AI program runs on a computer that belongs to the user or their company itself. This can be a server in the company’s basement, a rented machine in a data center, or even one’s own laptop. Ideally, the data never leaves the user’s own network.

Data protection, costs, and independence from the provider

The most common reason for this effort is the handling of sensitive data. A hospital may not simply send patient records to an outside service provider. Law firms, banks, and government agencies are also subject to strict rules. If the model runs in-house, the question of who else sees the data becomes much easier to answer. This is precisely why the topic is especially prominent in Europe.

A second reason is cost. With an external provider, you usually pay per request, often billed in tiny amounts per piece of text processed. This is very cheap for a few thousand requests per month. With millions of requests, the math can flip. Then owning your own hardware pays off, because its purchase price is a one-time expense, and usage afterward costs almost nothing.

Then there is independence. A provider can raise prices, discontinue a model, or replace it with a new version that behaves differently. Anyone who self-hosts decides entirely on their own when to switch. The technical term for this is vendor lock-in: dependence on a single supplier that is difficult to escape.

What technology is needed for this

At the start is a model whose blueprint and numerical values are publicly available. Such models are called open models; well-known examples are Llama from Meta, Mistral, or Qwen. You download a file that is several gigabytes in size. This file contains the parameters, i.e., the millions to billions of learned numerical values that determine the model’s behavior.

After that, suitable hardware is needed. Large language models compute on graphics cards, because these can perform very many simple calculations in parallel. The graphics card’s memory is decisive, since the model must fit into it completely. As a rough rule of thumb: a model with seven billion parameters takes up roughly four to fourteen gigabytes, depending on the memory format. Truly large models require several cards simultaneously, and a single professional card can quickly cost a five-figure sum.

Finally, between the file and the user lies software that loads the model and accepts requests. Common tools for this are called Ollama, vLLM, or llama.cpp. They provide an interface that feels almost the same to other programs as that of a commercial provider. An existing program can therefore often be switched over with just a few lines of code.

From hobby project to corporate IT

On a small scale, one encounters this topic in programs that run a language model directly on one’s own laptop. A current computer can handle smaller models without an internet connection, albeit more slowly and with weaker answers than large online services. For translations, summaries, or simple programming help, this is often enough.

On a large scale, it is a standard topic in corporate IT. Many companies operate their own models for internal search, document analysis, or support. The term appears in the news when government agencies publish tenders for sovereign AI infrastructure or corporations announce that they are bringing their AI into their own data center. A common misconception here: self-hosting does not automatically mean secure. A poorly secured in-house server can be more vulnerable than the data center of a large provider.