Autograd

At its core, an AI model is a very long computational formula with millions of adjustable numbers. These numbers determine what answer comes out in the end. For the model to get better, you need to know: if I increase this one number by a tiny bit, will the result get better or worse? This is exactly the question Autograd answers, and it does so for all numbers at once. The name is short for automatic computation of gradients. A gradient is nothing more than the answer to that question just posed: a number that indicates the direction and strength in which a given knob acts.

Why no one derives by hand anymore

The mathematics behind this question is called differentiation and has been known for centuries. For a short formula, you can work out a derivative with pen and paper. For a language model with hundreds of billions of knobs, that is completely out of the question. Before Autograd, researchers had to derive and program the derivatives themselves for every new network architecture. This was by far the biggest source of errors and slowed down every experiment.

Autograd did away with this step. You now only need to write down how the model computes. The derivatives are added automatically, exactly and without extra effort. As a result, someone can try out a new model idea within an hour that used to require weeks of preparatory work.

This acceleration is an underrated reason for the pace of AI development. It’s not just faster chips that have driven progress, but also the fact that experimenting has become cheap. Autograd itself is not a visible product, but a foundation on which everything else is built.

The computation path is retraced backwards

While the model is computing, Autograd keeps track of every single step. This creates a kind of log: first a multiplication happened, then an addition, then a function was applied. This log is called a computation graph. It is the map on which the way back is later traced.

At the end there is a single number, the error. It states how far the model’s output was from the desired answer. Starting from this number, Autograd runs the log backwards. For every computational step, it is known how it passes on an error. This way, the error is distributed step by step to all the knobs.

A comparison helps: imagine a chain of people passing along a whispered message. By the end, the message is garbled. You go through the chain backwards and ask each person how much they contributed to the error. This is exactly how Autograd distributes responsibility. This backward run is called backpropagation and is the reason why the procedure is so efficient: a single pass delivers the gradients for all knobs at once.

Autograd in PyTorch and in training costs

In practice, Autograd is mostly encountered in software libraries for machine learning. The best known is PyTorch, developed at Meta, whose core building block is actually called autograd. TensorFlow from Google and JAX also work on the same principle. Anyone training a neural network today uses Autograd, whether they know it or not.

The term rarely appears directly in the news, but its consequences do, constantly. When talk turns to enormous memory requirements during training, that is also due to the computation graph: all intermediate results must be kept until the backward pass needs them. That is why training consumes significantly more graphics card memory than simply running a finished model.

A common misconception is that Autograd is some kind of artificial intelligence in its own right. That’s not true. It is pure bookkeeping of derivatives, precise and stubborn. And it is only needed during training. When you use a chatbot, Autograd is switched off, because at that point the model isn’t supposed to learn anything anymore, just answer.