SeKV Cuts Long-Context GPU Memory 53% Without Discarding a Single Token

Every team running long-context inference eventually hits the same wall: the KV cache grows linearly with sequence length and never shrinks during decoding. The standard responses, token eviction and semantic grouping, both make the same structural mistake. They treat compression as a one-time decision made at prefill time, which means any token-level detail discarded early is gone permanently, even if a later query needs it.

SeKV sidesteps that trap entirely by storing context at variable resolution rather than choosing between keep and discard. The core idea is an entropy-guided segmentation pass that groups tokens into semantic spans, where spans with more predictable, redundant content get coarser representation and spans carrying higher information density get finer treatment. Each span then lives across two memory tiers simultaneously: a lightweight summary vector stays on GPU for fast routing, while a low-rank SVD basis for that span sits on CPU. During decoding, a trained zoom-in mechanism checks each query against the summary vectors and selectively pulls the SVD basis for relevant spans, reconstructing full token-level KV entries only where the current generation step actually needs them. The base LLM is completely frozen throughout. Fewer than 0.05% trainable parameters go into the zoom-in mechanism itself.

The analogy that makes the architecture legible: think of it as a map with variable zoom levels. The GPU holds the overview, enough to know which region matters. The CPU holds the satellite imagery for every region. The zoom-in mechanism decides which regions to render in full detail for each decoding step, without ever printing the entire map at once.

SeKV improves over the strongest semantic compression baseline by 5.9% averaged across four benchmarks, while cutting GPU memory 53.3% compared to full KV caching at 128K context. For ML infrastructure teams running long-context workloads, the takeaway is direct: the memory cost of 128K context is now negotiable without accepting the context-loss penalty that eviction-based methods impose.

We're thinking: We find the resolution-adaptivity framing more consequential than the memory number alone. The field has been treating KV cache compression as a binary choice, keep tokens or drop them, and optimizing around that constraint. SeKV reframes it as a storage-tier problem, which is a category of problem infrastructure teams already know how to solve. The specific implication worth watching: if zoom-in reconstruction quality holds at even longer contexts, the CPU tier effectively becomes a lossless KV extension, and the GPU becomes a routing layer. That architectural split could generalize well beyond LLM inference to any system where working memory costs exceed retrieval costs.

Key takeaways:

• SeKV organizes context into entropy-guided semantic spans stored across GPU and CPU at variable resolution, using a trained zoom-in mechanism to reconstruct token-level KV entries on demand during decoding, with the base LLM fully frozen.
• Across four benchmarks at 128K context, SeKV beats the best semantic compression baseline by 5.9% on average and cuts GPU memory 53.3% versus full KV caching; the caveat is that CPU-to-GPU reconstruction latency under high zoom-in rates has not been fully characterized at production serving throughput.
• Teams running long-context inference at 64K tokens or beyond should evaluate SeKV as a replacement for token eviction pipelines, particularly in settings where context recall precision matters and eviction-induced degradation has been a documented failure mode.

Source: SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory