State-Space Model

A state-space model is a particular architecture for learning computer programs that process data in a fixed order. Such data might be, for example, a text word by word, an audio recording, or measurements over time. The program works through the input step by step, keeping track of what is called a state: a fixed set of numbers that summarizes the most important information from what has been read so far. At each new step, this state is updated and the output is computed from it. The trick is that the state always stays the same size, whether ten or ten thousand words have been read. You can imagine it like a person reading a book who, after each page, revises their notes on a single sheet of paper instead of looking back through every previous page.

The way out of the length problem

Almost all well-known AI language models today are based on a different architecture, the transformer. There, the model compares every previous word with every other word again each time a new word is processed. This delivers very good results, but quickly becomes expensive. If you double the text length, the computational cost roughly quadruples. For very long inputs, this becomes a real cost problem.

A state-space model, by contrast, grows only linearly: twice as much text means roughly twice as much computation time. It also doesn’t need a growing intermediate memory for all previous words. This memory is often precisely the bottleneck in transformers when a chatbot needs to continue a long conversation. That’s why state-space models are considered a promising candidate for tasks involving very long inputs.

They are also interesting for small devices. A model that gets by with little memory can more easily run directly on a phone or in a car. This saves costs and protects data, because nothing needs to be sent to a remote server.

State, filters, and a bit of mathematics

The idea originally comes from control theory, that is, from the control of machines. There, a system is described by its state and by rules for how that state changes. One example is an autopilot that keeps track of an aircraft’s position and speed. The AI variant adopts this mathematics and lets the rules be learned from data instead of being set by hand.

Specifically, there are three kinds of number tables: one determines how strongly old information fades, one determines how new inputs flow into the state, and one determines how the output is derived from the state. Early versions had a problem here: information disappeared after just a few steps. Newer approaches such as S4 solve this through a cleverly chosen initial setting of these tables.

A further step came with the Mamba model in 2023. There, the rules depend on the current word. The model can therefore decide whether to incorporate something into the state or ignore it. A common misconception, by the way, is that state-space models are simply the old recurrent networks from the past. The difference is that training here can be structured in a form that modern graphics cards can process in parallel.

Where this technology appears today

In pure form, you rarely encounter state-space models in everyday life. Large chatbots like ChatGPT or Claude still rely predominantly on transformers. In trade press, however, the names Mamba, S4, or Jamba come up regularly when it comes to long contexts or faster models.

Hybrid forms, so-called hybrid models, have become widespread in the meantime. They combine many state-space layers with a few transformer layers. This gives you the speed of one architecture and the accuracy of the other. Several AI companies have already released such hybrids.

The technology is especially promising where data occurs in very long streams: in the analysis of gene sequences, in audio signals, in sensor data from factories, or in medicine. So if you read in a report that a model processes millions of characters at once, there is often a state-space component behind it.