Parameter

At its core, an AI model consists of a vast collection of numbers. These numbers are called parameters. At the start they are set randomly and the model can do nothing. Then it is shown many examples, perhaps millions of pages of text, and it changes the numbers until its predictions are mostly correct. This process is called training. What the model ultimately “knows” is stored exclusively in these adjusted numbers. When a report says a model has 70 billion parameters, this is exactly what is meant: 70 billion such adjustable values.

Why the parameter count appears in every model announcement

The parameter count is the simplest metric for comparing models. It roughly indicates how much a model can store at all. A model with three billion parameters can retain fewer relationships than one with 400 billion. That’s why manufacturers almost always mention the number in the name, as in “Llama 3 70B” – the B stands for Billion.

The number also determines the costs. Every parameter takes up memory and must be accounted for during computation. As a rough rule, a model needs about two gigabytes of memory per billion parameters. A model with seven billion parameters can therefore still run on a good laptop. A model with 400 billion parameters needs a data center with specialized graphics cards.

A common misconception is that more parameters automatically mean better answers. This only holds true given similar data quality. A smaller model trained on carefully curated texts regularly outperforms a larger one trained on poorer data. For several years now, the trend has therefore shifted toward more compact models that are nevertheless powerful.

How random numbers become knowledge

You can picture a model as a mixing board with billions of dials. Each dial determines how strongly a particular signal is passed on inside. A parameter is the position of such a dial. During training, a computational procedure automatically turns all the dials at once.

The process repeats millions of times, always in the same way. The model is supposed to predict the next word of a sentence. It makes a suggestion, and this is compared with the actual next word. From the difference, the system calculates the direction in which each individual parameter must be shifted so that the error becomes smaller. Each individual correction is tiny, but the sum of them ultimately produces a model that responds fluently.

It’s important to distinguish this from two other types of figures. Parameters are learned by the model itself. Settings such as the learning rate or the model size are determined in advance by humans; these are called hyperparameters. And the amount of training text is measured in tokens, i.e., word fragments – this is a figure about the data, not about the model.

Parameters in products and headlines

You encounter the number directly in model names. “Mistral 7B”, “Qwen 32B”, or “DeepSeek 671B” reveal the size right in the title. Anyone wanting to try out a model on their own computer chooses accordingly: models with three to eight billion parameters run locally, larger ones do not.

In business news, the parameter count is above all about money. Training large models costs hundreds of millions of dollars, mainly for chips and electricity. That’s why parameter counts often appear in the same paragraph as investments by Nvidia, Microsoft, or OpenAI.

It’s notable that the largest providers now often keep the exact number secret. For GPT-4 or Gemini, there are only estimates. With modern models, the figure is also less clear-cut than before, because some systems use only part of their parameters per request. In that case you’ll read two numbers: the total count and the number of actually active parameters.