Gradient Descent

When an AI model is trained, it makes many mistakes at the beginning. Gradient descent is the method that systematically reduces these errors. The model contains thousands or billions of numbers — called parameters — that determine how it responds to an input. Gradient descent changes these numbers repeatedly in small steps until the model improves. This doesn’t happen just once, but many millions of times in a row.

Why gradient descent is the heart of training

You can’t simply program a model to give the right answer. Instead, you show it examples and let it figure out on its own which parameters make the outputs correct. For this, you need a method that makes the error measurable and reduces it in a targeted way. Gradient descent is exactly this method.

Without such a method, finding the right parameters would be hopeless. A modern language model often has more than 70 billion parameters. Setting them all by hand is impossible. Gradient descent does this automatically — and that’s the reason AI can be trained at all.

Descending the error landscape

You can picture the training process as a mountain landscape. Every possible combination of parameters corresponds to a point in this landscape. The height of the point represents the model’s current error — the higher it is, the worse the model performs. The goal is to reach the valley, meaning to minimize the error.

At each step, the algorithm calculates the direction in which the slope drops most steeply. This direction is called the gradient — it indicates how strongly and in which direction the error changes when you shift a parameter. The model then takes a small step exactly opposite to this direction: downhill. How big this step is depends on what’s called the learning rate. A learning rate that’s too large can make the model overshoot the valley; one that’s too small makes the process extremely slow.

In practice, the method doesn’t check the error after every single training example, but after small groups — so-called mini-batches. This saves computing time and makes the learning process more stable. This variant is called stochastic gradient descent, or SGD for short, and is today the most common form of the method. Modern optimizers like Adam refine SGD even further by automatically adjusting the learning rate.

Gradient descent in products and headlines

Every time a company announces it has trained an AI model, gradient descent was running behind the scenes. GPT-4, Gemini, Stable Diffusion — all these models were trained using variants of this method. It is so fundamental that it’s often not even mentioned in technical articles anymore, because it’s taken for granted.

Gradient descent is also behind reports about training costs. Every single step of the algorithm costs computing time on specialized chips, so-called GPUs or TPUs. Because a training run involves billions of such steps, the costs of training large models can run into the millions of dollars.

A common misconception: gradient descent does not guarantee that the model finds the best possible parameters. The mountain landscape often has many local valleys — points that are lower than their surroundings but not the lowest point overall. The model can then end up in such a local minimum. In practice, this usually isn’t a big problem, because even a good local minimum results in a capable model.