Compressed Sparse Attention
Compressed Sparse Attention is a computational method in language models that partly summarizes and partly selectively focuses on long texts, instead of comparing every word with every other word. This allows programs like chatbots to process very long inputs without the computational cost exploding.
Programs like ChatGPT process text by checking which words belong together. In this comparison, the program in principle looks at every word together with every other word. For a short sentence, this is harmless. For a text of 100,000 words, this turns into ten billion comparisons. Compressed Sparse Attention is a shortcut for this computation. It merges distant parts of the text into rough summaries and only looks closely at a few, particularly important spots.
Why long texts are so expensive
The effort of the classical method grows quadratically. If you double the text length, the computing time quadruples. If you multiply it by ten, the effort increases a hundredfold. This is precisely why early chatbots had a narrow memory of only a few thousand words.
Today, however, users want to have entire books, legal texts, or program code analyzed in one go. A provider handling such requests pays for electricity and computing time for each individual one. Methods like Compressed Sparse Attention often lower these costs by a large multiple. This co-determines whether a service can even be offered affordably with long inputs.
A second point is speed. Nobody likes waiting two minutes for an answer. Whoever reduces the computational effort gets the answer back faster and can serve more users with the same hardware.
Summarizing and looking closely in a targeted way
The method combines two ideas. The first is compression: the text is divided into blocks of about 32 or 64 words. Each block is condensed into a single compact summary. Instead of a thousand individual words, the model then only needs to go through a few dozen block summaries.
The second idea is selection, called sparse in English. Summaries lose detail, and sometimes everything hinges on exactly one detail. That’s why the model uses the rough blocks to determine which few blocks are actually relevant right now. Only these are read out word by word again, while all others remain in their condensed form.
A comparison helps: You’re looking for a piece of information in a non-fiction book. You don’t read all 400 pages. You skim the table of contents — that’s the compression. Then you open three chapters and read them closely — that’s the selection. Importantly, the selection rules aren’t set by humans but are learned along the way during the model’s training.
Where the technology is found in products
The method becomes visible in the specification of the context window, i.e., the maximum amount of text a model can take in at once. When a provider advertises a million tokens, such sparsity tricks are almost always behind it. Without them, such a long input would be technically almost unaffordable.
In everyday life, you encounter the result when you upload a PDF file into a chatbot, when a coding assistant keeps an overview of an entire project, or when a model retains the summary of a long conversation history. In trade news, the term usually appears alongside related methods, such as Native Sparse Attention from DeepSeek or Sliding Window Attention.
A common misconception is that this makes the model smarter. That is not the case. It becomes faster and cheaper, and in an unfavorable case, the selection may overlook an important spot. Good implementations achieve nearly the same quality as the full method on the usual benchmarks, at a fraction of the effort.