Muon

An AI model consists of many billions of numbers, known as weights. During training, these numbers are repeatedly adjusted a little until the model delivers good answers. The rule by which this adjustment is calculated is called an optimizer. Muon is such an optimizer, developed in 2024 by Keller Jordan and other researchers. It differs from previously common methods in that it does not consider each number individually, but treats entire number tables as a unit. The name is a made-up word combining “Momentum Orthogonalized by Newton-Schulz.”

What a faster optimizer saves in computing time

Training large language models costs double-digit millions, often over weeks on thousands of graphics cards. Whoever achieves the same model quality with fewer computing steps saves money and electricity directly. That is exactly what Muon promises: in published comparisons, it needed roughly a third to half less computing effort for the same result than the previous standard. That is an unusually large leap in the field of optimizers.

For more than a decade, a method called Adam was practically unrivaled. Dozens of supposed successors failed to deliver on their promises in independent tests. Muon is one of the few that proved convincing even outside its own lab. That explains the attention the optimizer has received in a short time.

An important qualification: Muon does not make models smarter than their size allows. It merely shortens the path there. For companies, this is nevertheless decisive, because with the same budget they can process larger models or more training data.

The idea of the balanced number table

A model’s weights are organized in tables, so-called matrices. Older optimizers like Adam treat each entry of such a table individually. Muon, by contrast, looks at the proposed change as a complete table. It finds that this change is usually very uneven: it points strongly in a few directions, but barely in many others.

Muon balances this out. It recalculates the change table so that all directions are weighted roughly equally. This step is called orthogonalization. A comparison: instead of revising a text only at its most conspicuous spot, you improve all paragraphs to a similar degree. Small but useful learning directions are thus not lost.

This recalculation would normally be very expensive. Muon therefore uses an approximation method called the Newton-Schulz iteration, which manages with simple multiplications in just a few steps. Graphics cards are built for exactly this kind of computation, so the additional overhead remains small. For parts of the model that do not have a table form, such as the word vectors at the input, Adam continues to be used. Muon is therefore not a complete replacement, but a tool for the main part of the network.

From Kaggle records to Kimi K2

Muon first became known through so-called speedrun competitions. There, developers try to train a small language model to a fixed quality in as little time as possible. Muon lowered these best times significantly and then spread quickly through the research community.

The biggest real-world test was delivered by the Chinese company Moonshot AI. It trained its language model Kimi K2, with around one trillion parameters, using an adapted variant called Muon Clip. This proved that the method remains stable even at a very large scale. This was exactly where earlier Adam alternatives had often failed.

In everyday life, you do not encounter Muon directly. The optimizer works during the training process and is no longer part of the finished chatbot. However, the name regularly appears in trade news and model reports, usually as an indication of how efficiently a model was created.