Mamba-2

Mamba-2 is a blueprint for programs that process language. Such programs read a text and predict which word fits next. Most well-known systems of this kind do so by comparing every word with every other word in the text. This becomes very expensive once the text gets long: doubling the length quadruples the effort. Mamba-2 takes a different approach. It reads through the text from beginning to end while keeping a kind of notepad that always stays the same size. The architecture was published in 2024 by researchers from Princeton and Carnegie Mellon universities, as the successor to the one-year-older model Mamba.

What the fixed notepad means for long texts

The computational effort is really the crux of the matter. A typical language model has to scan the entire preceding text again every time it writes a new word. The longer the conversation, the slower and more expensive each additional reply becomes. Mamba-2, by contrast, only looks at its notepad. That notepad is exactly the same size after ten words as it is after a hundred thousand.

In practice, this means such a model can process an entire book, a long piece of program code, or an hours-long conversation transcript without costs exploding. This is precisely what many companies are currently working on, since users keep pasting ever longer documents into chatbots. Memory requirements during response generation also stay constant, which makes small devices like phones an interesting use case.

The price for this is a drawback in precise recall. Because everything read gets compressed into a notepad of fixed size, detail gets lost. If the model is supposed to reproduce a specific number from page 300 word for word, the classic architecture has an easier time of it. That’s why many current models combine both approaches.

State, selection, and the trick with matrices

The notepad is technically called the state. With every new word, two things happen: the old state fades a bit, and new information gets blended in. How strongly it fades and how strongly it gets blended in is decided anew by the model for every word. This capability is called selectivity, and it was the main innovation of the original Mamba. As a result, the model can almost ignore unimportant filler words, while important names are retained for a long time.

The central idea behind Mamba-2 is called State Space Duality, or SSD for short. The researchers showed that this step-by-step reading process is mathematically the same as a large multiplication of number tables, so-called matrices. Such multiplications are exactly what graphics cards are built for. During training, one can use the matrix form and process the entire text in parallel. When generating responses, one uses the step-by-step form and saves memory.

To make this fast, Mamba-2 gives up some flexibility compared to Mamba-1: the state is more simply structured, but noticeably larger, and the training loop runs, according to the authors, two to eight times faster. A common misconception, by the way, is that Mamba-2 is a ready-made chatbot. It is only the architecture, i.e. the blueprint — the knowledge only comes about through training on text data.

Mamba-2 in real models and in the news

In products, one rarely encounters Mamba-2 in pure form, but usually as a mixture. In such hybrid models, only every fourth or fifth layer is a classic comparison layer, while the rest works on the Mamba principle. Well-known examples are Jamba from the Israeli company AI21 Labs, the Nemotron models from Nvidia, and Codestral Mamba from the French provider Mistral. These models explicitly advertise very long input windows.

In trade news, the name usually comes up in connection with keywords like context length, inference costs, or alternatives to the Transformer. When a company announces that its model can process hundreds of thousands of words at low cost, such an architecture is often behind it. Anyone who understands the basic idea also understands why such announcements can be realistic.

For you as a user, the difference is invisible. At most, you notice it in that a model still responds quickly even after long conversations, or that it runs on a laptop without an internet connection. Whether the result is better in terms of content still depends mainly on the training, not on the blueprint alone.