Sparse Attention
Sparse Attention is a cost-saving technique in modern language models: instead of comparing every word in a text with every other word, only selected word pairs are considered. This drastically reduces computational effort for long texts and makes inputs of hundreds of thousands of words affordable in the first place.
Programs like ChatGPT process text by relating each word to the other words. Only this way does it become clear what a “he” or a “that” refers to. Normally, the program truly compares every word with every other word to do this. For a short sentence this is effortless, but for an entire book it becomes enormous: with twice as many words, there are four times as many pairs. Sparse Attention is the answer to this. The program only compares a deliberately selected portion of the word pairs and leaves out the rest. “Sparse” describes exactly that: the table of comparisons has large empty areas.
Why long texts would otherwise be unaffordable
The effort grows quadratically. For 1,000 words, that’s around one million comparisons. For 100,000 words, it’s already ten billion. This is exactly why early language models had a limit of a few thousand words per input. Anyone who wanted more had to break the text into chunks and lose connections across the interfaces.
Today, providers advertise that you can feed in entire code projects, court files, or books all at once. These large input windows of hundreds of thousands or even millions of words are practically unaffordable without cost-saving techniques like Sparse Attention. Every request costs computing time on expensive specialized chips, and these costs are incurred anew with every single use.
Then there’s memory. The model must retain intermediate results for every word already processed, so that it doesn’t have to compute from scratch for the next word. For very long texts, this cache fills up graphics card memory faster than the model itself does. Sparse Attention also reduces this need, because fewer old words are queried at all.
Which word pairs remain
The simplest variant is a window: each word only looks at its nearest neighbors, say the last 500 words. This suffices for grammar and local coherence, but loses the connection to the beginning of the text. That’s why the window is usually combined with a few global anchors. These are words that everyone is allowed to see and that themselves are allowed to see everything, something like a bulletin board for the entire text.
A good analogy is a corporate network. Everyone talks daily with their team, and there are also a few people at headquarters with whom everyone stays in contact. No one has to know every colleague individually, and yet information reaches the whole company in just a few steps. Some methods additionally scatter in random connections, so that unexpected connections also get a chance.
Newer models no longer fix the pattern but learn it. A small auxiliary network estimates in advance which sections of text are even relevant for the current word, and only those are checked precisely. This is more accurate than a rigid window, but harder to program, because graphics cards work fastest when processing uniform blocks. That’s why most implementations don’t compute with individual words but with blocks of about 64 words.
Where the technology is found in products
You rarely encounter Sparse Attention under this name, but rather through its consequences. When a provider advertises a context window of a million characters, such a cost-saving technique is almost always behind it. Coding assistants that can survey an entire project directory also work this way. In technical articles, names like Longformer, BigBird, or Sliding Window Attention show up, all of which are variants of the same idea.
A common misconception is that Sparse Attention is the same as Mixture of Experts. Both save computing time, but at different points. Mixture of Experts switches off parts of the model, Sparse Attention cuts comparisons between words. Large models often use both at the same time.
The price is a quality risk. If an important connection happens to fall through the grid, the model overlooks a relationship. For tasks where a single detail from page three matters on page 800, sparse models therefore sometimes perform worse. Providers test this with tasks in which a single sentence is hidden within a huge text.