Jacobian Matrix

Imagine a machine that has several dials and several displays. You turn one dial a tiny bit. Each display reacts to this with a different strength, some not at all. The Jacobian matrix is exactly the table that collects all these reactions: each row represents a display, each column a dial. Each entry states how sensitively that display reacts to that dial. It is named after the mathematician Carl Gustav Jacob Jacobi, who worked in Königsberg and Berlin in the 19th century.

Why learning doesn’t work without it

At its core, a neural network is a huge computational rule with a great many dials. These dials are called parameters or weights. During training, one wants to set them so that the outputs come as close as possible to the desired answers. For this, one needs to know in which direction each individual dial should be turned. It is exactly this information that the Jacobian matrix provides.

Without it, only blind trial and error would remain. For a model with billions of parameters, this would be hopeless. The matrix turns this into a directed procedure: after every computational step, one knows where things go downhill. That is why it is not mathematical decoration, but the reason modern AI is trainable at all.

It is important to distinguish it from the gradient. A gradient describes the same principle, but only for a single output number. The Jacobian matrix is the generalization to many outputs at once. A gradient is thus a Jacobian matrix with exactly one row.

How a computer calculates it

One could calculate every sensitivity individually by hand. With billions of entries, that is impossible. Instead, one uses automatic differentiation. In this process, the computer breaks the large computation down into many tiny steps such as addition or multiplication. For each of these steps, the sensitivity is trivially known.

These small partial results are then combined with one another backwards through the network. This going-backwards is called backpropagation. It is the standard way in which program libraries like PyTorch or TensorFlow operate. The user only writes down the forward computation, and the derivatives arise automatically.

In practice, the full matrix is almost never assembled. It would simply be too large for any memory. Instead, one only computes its product with a vector, i.e. the effect in a particular direction. This suffices for training and is cheaper by orders of magnitude. A typical misconception is therefore to imagine the Jacobian matrix as a table that actually exists in memory.

From robot joints to image generation

In robotics, the Jacobian matrix is an everyday tool. A robotic arm has several joints, and the gripper hand is supposed to reach a specific point in space. The matrix translates between joint angles and hand position. It answers the question of how fast the hand moves when a joint rotates.

In image processing, it appears when images are distorted or rotated. It also plays a role in models that generate images. Some image-generation methods need to calculate how strongly a transformation compresses or stretches space. For this, one uses the so-called determinant of the Jacobian matrix, a single number for the stretching factor.

In trade news, you tend to encounter the term rather indirectly. When there is talk of gradients that explode or vanish, this concerns properties of such matrices. Reports about faster training methods, too, usually concern the question of how to obtain these derivatives more efficiently. Anyone who knows the term understands such headlines considerably better.