Gradient

A learning computer program has many adjustable dials it can turn. At the start, these dials are set randomly, and the program accordingly produces incorrect results. One therefore measures the error: a single number indicating how far off the output is. The gradient is the answer to the question of how this error changes when one turns a dial by a tiny amount. It states, for every single dial, both direction and strength: up or down, a lot or a little. Mathematically it comes from differential calculus, but it can be understood without this background knowledge.

The engine behind every training run

Without gradients, one would have to set the dials by guessing. For a model with a few dozen dials, this would be tedious but feasible. Modern language models have hundreds of billions of them. Random trial and error would take longer than the universe has existed.

The gradient solves this problem because it treats all dials simultaneously. A single computation yields the appropriate correction for each of them. As a result, the effort does not grow explosively with model size but remains manageable. This is precisely what has made very large models possible in the first place.

When news reports state that a training run cost millions, what lies behind this is above all one thing: the billions-fold computation of gradients on specialized graphics chips. The gradient is thus not merely a mathematical detail but the largest cost item in the AI industry.

Walking downhill in the fog

A common analogy is a hilly landscape in dense fog. Height represents the error, and each point in the landscape represents a particular setting of all the dials. The goal is to find the deepest valley. One cannot see anything, but one can feel in which direction the ground slopes most steeply. This direction is the gradient.

Training turns this into a simple recipe. One feels out the slope, takes a small step downhill, and repeats this. This procedure is called gradient descent. How large the steps are is determined by what is known as the learning rate, a value chosen by humans. Steps that are too large jump over the valley, while steps that are too small take forever.

Gradients are computed using a procedure called backpropagation. In this process, the error is traced backward through the network starting from the result. This way, each layer is assigned its share of the blame. A common misconception is that the gradient indicates where the goal lies. It only indicates which way leads downhill from here, and that is all the procedure needs.

Where gradients show up in the news

Gradients themselves are rarely directly visible, but their side effects are. A well-known problem is vanishing gradients: in deep networks, the values become ever smaller on the way back, until the earlier layers learn almost nothing anymore. Conversely, there are exploding gradients, where the values become so large that training derails. Both are standard topics when experts explain why a training run fails.

Gradients also play a role in data privacy. In federated learning, data stays on the phone, and only computed gradients are sent to a server. Research has shown, however, that original data can sometimes be reconstructed from these numbers. Such reports regularly appear in security reports.

In practice, anyone who trains an AI themselves encounters this term. Software libraries such as PyTorch compute gradients automatically, visible through commands like backward. And those who merely use models benefit indirectly: every response from a chatbot is based on settings that emerged from millions of small steps taken downhill.