Mamba

Mamba is a blueprint for programs that process language. Such programs receive a text and predict which word fits next. To succeed at this, they must remember what came before in the text. Mamba solves this remembering in a particular way: the model reads the text from front to back and keeps a small notepad as it goes. After each word, the notepad is updated, but it never grows longer. The design was introduced in 2023 by researchers from Carnegie Mellon and Princeton Universities and has since received a great deal of attention.

The dispute over the Transformer’s dominance

Almost all well-known AI language models are based on a design called the Transformer. When processing, a Transformer compares each word with every other word in the text. This works remarkably well, but it has an expensive side effect. If you double the text length, the computational effort quadruples. With a single sentence, nobody notices anything. With a 500-page book, it becomes a serious problem.

Mamba, by contrast, grows only linearly: twice as much text means roughly twice as much computing time, not four times as much. For very long inputs, Mamba is therefore many times faster than an equally sized Transformer. Memory requirements are just as important. A Transformer must temporarily store all previous words when generating a response, whereas Mamba only needs to store its notepad of fixed size.

This is interesting wherever long data sequences occur. That includes genomes with millions of building blocks, audio recordings, or sensor readings. For the industry, there is also a fundamental question at stake. Is the Transformer really the only option, or are there cheaper ways to reach the same goal?

The notepad and what ends up on it

Technically, Mamba belongs to the family of state space models. The technical term sounds complicated, but it refers exactly to the notepad described above. The state is a fixed set of numbers that summarizes the text so far. For each new word, the model computes a new state from the old state and the word.

Older models of this kind treated every word the same and therefore performed poorly on language. Mamba’s decisive innovation is called selectivity. The model decides anew for each word how strongly it changes the state. A filler word like “and” can pass by almost without a trace. A name or a number can rewrite the state forcefully and thus remain preserved for a long time.

All of this would be worthless if training ran too slowly. A sequential pass fits poorly with graphics cards, which handle thousands of calculations simultaneously. The Mamba developers therefore wrote special programs that keep the state in the graphics card's fast cache memory. One well-known drawback remains, however: because the notepad never grows, Mamba cannot retain every detail verbatim. On tasks that require exact quoting from a long text, Transformers are often ahead.

Mamba in real products and news

In headlines, Mamba usually appears when a lab announces a new model with a particularly large context window. The context window is the amount of text a model can take in at once. The Israeli company AI21 released Jamba, one of the first large models to mix Mamba blocks and Transformer blocks. Nvidia and Mistral have also introduced such hybrid models.

This mixing is currently the usual approach. Pure Mamba models have remained rare in practice. Developers prefer to combine a few Transformer layers for precise lookup with many Mamba layers for speed. Anyone reading model names often recognizes such designs by the word component “hybrid”.

As a user, you don’t notice the design directly. It shows up indirectly in how much text a service accepts and what it costs. One common misunderstanding, by the way, is a mix-up of names: Mamba has nothing to do with the Mamba software for package management in programming languages. Both names merely come from the same snake.