SWA-based HiCache

When a language model writes text, it generates one word after another. For each new word, it looks at everything that came before. So that it doesn’t have to repeat this groundwork every time, the server stores the intermediate results in a temporary store called a cache. In long conversations, however, this store becomes huge and no longer fits on the fast graphics-card chips. SWA-based HiCache solves this with two ideas at once: First, the model considers only a limited window of the most recent words per word instead of the entire text. Second, the intermediate results are distributed across several memory tiers, ranging from very fast and scarce to slow and cheap.

What limited memory costs in long chats

The fast memory of an AI graphics card is expensive and small, often 80 to 192 gigabytes. A single long chat with hundreds of thousands of words can occupy several gigabytes of it. If many users are active at the same time, the memory fills up before the compute capacity is exhausted. The server then has to make requests wait or discard old intermediate results.

Discarded intermediate results are particularly annoying. If the user comes back with a follow-up question, the server has to recompute the entire conversation so far. For very long texts, the first answer then takes many seconds. This waiting time for the first word is one of the most important metrics for the quality of an AI service.

Both building blocks work together here. The sliding window reduces the amount of data per conversation, while the memory hierarchy creates additional space for many conversations at once. In practice, this allows significantly more users to be served on the same hardware. For operators, this is directly a matter of cost per answer.

Sliding window plus memory tiers

SWA stands for Sliding Window Attention. Instead of considering all previous words, the model in such layers only sees the last few thousand. The window moves along with the text, like a reading slot over a long paper scroll. Older words fall out of the window, and their intermediate results can be deleted. As a result, memory demand no longer grows with text length but stays roughly constant.

HiCache stands for hierarchical caching. The intermediate results are stored not only in graphics-card memory but also in the server’s regular working memory and, if necessary, on SSD drives. At the top it’s fast and scarce, further down it’s slow and spacious. Whatever is currently needed moves upward, whatever has been resting longer sinks downward. The principle is the same as with a desk: current sheets lie in front of you, older ones on the shelf, very old ones in the basement.

Many models mix the two types of attention. Some layers work with the sliding window, while a smaller portion continues to see the entire text. This preserves the ability to remember a number from the beginning of the document. It is precisely these full layers whose data is then captured by the memory hierarchy. A common misconception is that the sliding window limits the maximum text length. It only limits how far back a single layer directly looks.

Where the technique shows up in practice

You will mostly encounter this term in the documentation of inference servers, i.e., the software that makes finished models available for requests. Projects such as SGLang or vLLM explicitly advertise hierarchical caching and support for Sliding Window Attention. Announcements then feature figures on throughput and on the waiting time for the first word.

On the model side, sliding windows appear in many open model families, such as Mistral or Gemma. Anyone running such models themselves almost automatically runs into the cache settings during setup. Large providers also use related methods, but prefer to speak of prompt caching and offer discounts on reused text inputs.

As a user, you only notice the effect. A chat with a long uploaded document answers follow-up questions faster than the first time. And services can offer long contexts at affordable prices at all. That is exactly why such caching techniques are regularly a topic in trade news.