Backbone Parameters
Backbone parameters are the adjustable numbers in the main body of an AI model that carries the general foundational knowledge. They emerge during expensive base training and mostly remain unchanged during later adaptations.
At its core, an AI model consists of a great many numbers that are repeatedly readjusted during learning. These numbers are called parameters, and they determine what the model does with an input. Most of them sit in the main part of the model, which is needed for every task. This main part is called the backbone. The numbers within it are the backbone parameters. They store the model’s general capabilities: language sense, world knowledge, the recognition of shapes in images. On top of this sit smaller parts that are tailored to a specific task.
Why the backbone is so expensive
The backbone parameters make up almost the entire weight of a model. In a large language model, this quickly amounts to 70 billion or more numbers. Training them from scratch once costs millions of euros in computing time and electricity. This is exactly why there are only a few companies worldwide that actually build new backbones.
Everyone else continues to use an existing backbone. This is called transfer learning: one takes over the expensively learned base knowledge and only adapts a thin layer on top of it. A start-up building a legal advice AI, for example, does not train language understanding from scratch. It builds on an existing backbone and thereby saves almost all of the costs.
This also explains why openly available models are so economically important. Anyone allowed to download a strong backbone for free receives, practically as a gift, what would otherwise require a data center. Many debates about open versus closed AI models revolve around exactly this question.
Frozen or trained along
When adapting to a new task, there are two options. In the first, one freezes the backbone parameters. Frozen means: they are no longer changed during further training. Only the upper, task-specific parts continue to learn. This is fast and also works with just a few thousand examples.
In the second option, one lets the backbone learn along as well, usually with very small learning steps. This often delivers better results but is risky. If the parameters are changed too strongly, the model forgets part of its previous capabilities. Experts call this catastrophic forgetting.
As a compromise, methods such as LoRA have become established. Here, the backbone remains unchanged, and one merely places small additional matrices alongside it that shift the behavior. Instead of billions of parameters, one then trains perhaps a few million. This fits on a single graphics card instead of a data center.
Where the term appears in reports
It is most commonly read in image processing. Models such as ResNet or Vision Transformer are explicitly referred to there as backbones. On top of this, one then places a so-called head, which, for example, marks objects in photos or sorts images into categories. One backbone, many heads, many applications.
In language models, one more often speaks of the base model or foundation model. This means practically the same thing: a broadly trained backbone that is later refined. When a company announces that it has specialized a model for its industry, it has, as a rule, taken over the backbone unchanged.
A common misconception: the number of backbone parameters alone says little about quality. A well-trained model with 8 billion parameters can outperform a poorly trained one with 70 billion. What matters is the quantity and quality of the training data as well as the architecture. The parameter count is merely a size indication, not a grade.