Adam Optimizer
The Adam optimizer is a computational method that, during the training of AI models, determines how strongly the internal parameter values are changed after each learning step. It adjusts the step size individually for each value and has been the standard method for training neural networks since 2015.
An AI model consists of millions or billions of numbers that are randomly set at the start. For the model to do something meaningful, these numbers must be improved step by step. After each attempt, one measures how wrong the output was and changes the numbers a small amount in the better direction. An optimizer is precisely the method that decides how large this step turns out to be. The Adam optimizer is by far the most commonly used variant of this today. Its name comes from “Adaptive Moment Estimation.”
Why almost every large model is trained with Adam
Before Adam, a simpler method was mostly used: plain gradient descent. There, the same step size applies to all numbers in the model. This is a problem because some numbers have a strong effect on the result while others have almost none. If the step size is chosen large, the sensitive values jump wildly back and forth. If it is chosen small, the sluggish values take forever.
Adam solves this by giving each individual number in the model its own, continuously adjusted step size. This means less manual tuning is required from humans. A training run that with the old method would only have worked after days of trial and error often runs successfully with Adam on the first attempt.
The practical effect is enormous. A training run for a large language model can easily cost several million euros in computing time. A method that reaches its goal faster and fails less often thus saves real money. The 2015 research paper on Adam is among the most-cited works in all of computer science.
Momentum and brake: Adam’s two memories
Adam remembers two things from the past for each number in the model. The first is the average direction of the most recent changes. This acts like momentum: if the last steps all pointed in the same direction, progress continues there more briskly. One can picture it like a ball rolling down a slope. It picks up speed instead of starting anew at each step.
The second memory stores how strongly the changes fluctuated recently. A number with large, erratic swings gets a smaller step size. A number that has barely moved for a long time gets a larger one. This is the brake that prevents sensitive values from overshooting the target.
From both memories, Adam calculates its own update value for each number. The human only specifies a base step size, the learning rate. A common misconception is that Adam makes this learning rate superfluous. It remains the most important value to tune; Adam merely distributes it more cleverly across the individual numbers.
AdamW, training curves, and what shows up in technical papers
Anyone programming a neural network, whether at school or at a corporation, usually chooses Adam as the default. In the common programming libraries, it is available as a single line of code. Image generators and translation models have also almost invariably been trained with it.
In technical reports on new models, one often finds the abbreviation AdamW. This is a slightly modified version that additionally prevents the numbers in the model from becoming unnecessarily large. Large language models today are typically trained with AdamW, not with the original from 2015.
A drawback appears with very large models. Adam must keep two additional values in memory for each number. This roughly triples the memory requirements of training. This is precisely why research is being conducted into finding more memory-efficient methods. Until then, Adam remains the standard against which every new development must be measured.