Weight (neural network)
A weight is a single number in an AI model that determines how strongly a signal is passed on to a next point. All weights together form a model's entire knowledge; they are readjusted millions of times during learning.
A program that understands language or recognizes images consists at its core of a great many numbers. These numbers are called weights. Each weight indicates how strongly a particular signal is passed on within a chain of computations. A large weight amplifies the signal, a weight near zero suppresses it almost entirely. A negative weight reverses the effect: the signal then works against a result. You can imagine weights as knobs on a gigantic mixing console that has millions or billions of such knobs.
Why all the knowledge resides in the weights
A newly created model has random weights and produces nonsense. The learning process changes nothing other than precisely these numbers. Afterward, the model can translate, write code, or mark tumors on X-ray images. The learned knowledge, then, does not sit in a database of sentences but is distributed across the numerical values.
That is why the number of weights is one of the industry’s most important metrics. They are usually called parameters. When news reports speak of a model with 70 billion parameters, they mean 70 billion such numbers. More weights roughly mean more capacity, but also more memory requirements and higher costs per response.
For companies, the finished weights are the actual capital. Training often costs many millions of euros in compute time and electricity. Whoever owns the weight file owns the result of that investment. This is exactly why companies argue over whether to publish their weights or keep them secret.
How the numbers find their values
Training proceeds in rounds. The model is given an example, say half a sentence, and must predict the next word. A computation step then compares the prediction with the correct answer. The deviation is called error or loss.
The procedure then calculates, for each individual weight, in which direction it would make the error smaller. Each weight is shifted a tiny bit in that direction. This backward calculation is called backpropagation. A single step accomplishes almost nothing, but there are billions of such steps.
A common misconception is that a single weight would be responsible for a recognizable concept. That is almost never the case. Meaning arises from the interplay of many weights, and a single value alone says nothing. This is why it is so difficult to explain a model’s decisions after the fact.
Open weights, downloads, and storage space
Whoever downloads a model from the internet is, in reality, downloading a file containing the weights. Such files are large: for a model with seven billion parameters, several gigabytes are typical depending on the storage format. The program that computes with them is, by comparison, tiny. Without the weights, it is an empty shell.
In the news you hear the term “open weights.” This means a company makes the weights freely available for download. This is not the same as open source, since the training data and the training program usually remain secret. You get the result, not the recipe.
Weights can also be altered afterward. During fine-tuning, a finished model is trained further with one’s own examples, for instance a company’s manuals. During quantization, the same weights are stored more coarsely so that they fit on an ordinary laptop. In both cases the core remains the same: you work on the numbers, not on the program code.