Model Size

An AI system like a chatbot consists at its core of a huge table of numbers. These numbers are called parameters. They are not entered by humans, but are automatically set during learning from example texts. Model size is simply the answer to the question: How many such numbers are there? Common figures are “7B” or “70B”, where B stands for billion. A model with 70B thus has around 70 billion adjustable numbers inside.

What the parameter count reveals about a model

Model size primarily determines what operation costs. Every single number must reside in memory while the model is working. As a rough rule of thumb, a model needs about two gigabytes of memory per billion parameters. A 7B model thus fits on a good graphics card in a home PC. A 400B model needs an entire data center with many specialized chips.

For a long time, the rule was: bigger is better. Indeed, experiments from around 2020 onward showed that models with more parameters reliably delivered better results. This observation is called scaling laws. It explains why companies competed for years with ever-larger models.

By now the picture is more complicated. A small model trained on very large amounts of very clean text often beats a larger one with worse training material. Model size alone is therefore no seal of quality. It is more like the displacement of an engine: informative, but not the whole car.

Where the parameters sit within the model

A language model is built from layers that are passed through one after another. Each layer contains large tables of numbers through which the text is computed. Model size results from how many layers there are and how wide each one is. Developers fix both of these before training. Afterwards, the size can no longer be changed.

An important distinction concerns active parameters. Some models use only a fraction of their numbers for each request. This principle is called Mixture of Experts: a small additional network selects which parts of the model actually do the computing. Such a model may have 600 billion parameters but activate only 30 billion per response. It then computes as fast as a small model, yet still needs the storage space of a large one.

In addition, size can be reduced afterward. In so-called quantization, each number is stored more coarsely, for instance with four instead of sixteen digits. The model shrinks to a quarter of its size and loses only a little accuracy. Another method, distillation, has a small model imitate the answers of a large one.

How size figures appear in product names

Model names almost always contain the size directly. Examples are “Llama 3 8B” or “Mistral 7B”. Anyone reading such names can immediately estimate where the model can run. Anything below about 10B runs on good consumer hardware, and in some cases even on newer smartphones.

For commercial systems like ChatGPT or Gemini, providers keep the model size secret. It is considered a trade secret because it allows conclusions to be drawn about costs and design. Reports in business media therefore often contain only estimated figures.

In the news, the term usually appears in connection with money and energy. When talk turns to billion-dollar investments in data centers, it is almost always about operating very large models. At the same time, there is a countertrend toward small models that run directly on the device. They are cheaper, faster, and do not send data to foreign servers.