Linear Attention
Linear Attention is a computational method used in language models in which the effort grows only proportionally to text length instead of quadratically. This makes it possible to process very long texts without computing time and memory requirements exploding.
Modern language programs don’t read a text word by word in isolation. For each word, they check which other words in the text are relevant to it. This comparing process is called attention. In the standard procedure, every word is compared with every other word. With 1,000 words, that’s a million comparisons; with 10,000 words, it’s already a hundred million. Linear Attention is a group of computational tricks that changes this behavior: the effort then grows only at the same rate as the text itself, no longer quadratically.
What makes long texts so expensive
The difference between quadratic and linear growth sounds technical, but it determines the costs. If you double the text length, the effort doubles under linear growth. With the quadratic procedure, it quadruples. If you increase the length tenfold, the factor is a hundred. Beyond a certain text length, the standard procedure simply becomes unaffordable.
This is exactly where one of the major limitations of today’s language models lies. A model is supposed to summarize an entire book or search a company database. It’s supposed to remember everything said hours earlier in a conversation. This amount of text is called the context window. The larger the window is meant to be, the harder the quadratic calculation hits.
Then there’s memory. The standard procedure stores intermediate results for every word read so far, the so-called caching of keys and values. This memory grows with every additional word. Linear attention methods instead maintain a state of fixed size. Whether the text is 1,000 or 100,000 words long makes little difference to the memory requirement.
The computational trick with the order of operations
Attention works with three lists of numbers per word. They are called query, key, and value. The query says, in effect: this is what I’m looking for. The key says: this is what I offer. The value contains the actual information that gets passed along. In the standard procedure, every query is computed against every key, and this produces the large quadratic table.
Linear Attention avoids this table by rearranging the order of the computational steps. Instead of first comparing queries with keys, keys and values are first combined into a kind of collective folder. This folder has a fixed size. Each new query then only needs to access the folder once. Mathematically, this is a regrouping within a multiplication, similar to placing parentheses differently in a calculation to save work.
For this restructuring to be permissible at all, one must forgo a computational function that, in the standard procedure, weights the comparison values. It is called softmax and ensures that the model can focus very precisely on individual words. Linear Attention replaces it with simpler approximations. This is the price: attention becomes somewhat less sharp. On tasks where a single detail from a long text must be retrieved exactly, pure Linear Attention models often perform worse.
Where the technology appears in products
In technical articles and company announcements, Linear Attention usually comes up in connection with very large context windows. When a provider advertises that its model can process a million words at once, such a cost-saving method is almost always behind it. Related names that appear in the same announcements are Mamba, RWKV, or state space models in general. They are based on the same fundamental idea of a running state of fixed size.
In practice, many providers rely on hybrid forms. Part of the model’s layers compute classically and precisely, another part linearly and cheaply. This way, precision is preserved for short passages of text, while costs remain manageable for long texts. Such hybrids are currently considered the most practical approach.
Users notice this in quite everyday things. A chatbot still responds quickly even after a long conversation. An assistant can read in an entire PDF document instead of just excerpts. And language models are increasingly running directly on phones, where memory is scarce. One common misconception, by the way, is that Linear Attention makes models smarter. Above all, it makes them cheaper and more enduring when handling long texts.