Key takeaways
- A KV cache saves the attention keys and values of every token already processed, so each new token needs only its own to be computed.
- Its size grows linearly with context length and is kept per request, which limits how many requests a server can batch.
- PagedAttention in vLLM cuts wasted cache memory, and methods like LowRAM shrink the cache itself; the LowRAM authors report up to 26.9x compression at 94.5% of baseline performance in one setup.
A KV cache is the memory a language model keeps of the text it has already read, so it does not have to redo that work for every new word it writes. "KV" stands for key and value, two vectors that the model's attention layers compute for each token. The cache holds them, and it grows with every token of context, which is why it is a main limit on how many long conversations a server can handle at once.
The idea is old and the cost is now well understood. A paper posted on November 4, 2026 puts it plainly: the cache grows linearly with context length, which limits batch size and throughput. This article explains what is in it, why it gets big, and what engineers do about that.
What a KV cache stores
A transformer reads text as tokens. In the original Attention Is All You Need paper, attention is described as mapping a query and a set of key-value pairs to an output. For each token, each attention layer produces one query, one key and one value. The query asks what to look for. The keys say what each earlier token offers. The values carry the content that gets mixed into the answer. The paper's formula is softmax(QKᵀ/√d_k)V.
When the model writes text, it produces one token at a time. The paper's decoder is built so that a position cannot look at later positions, which keeps generation autoregressive: each new token depends only on the ones before it. That has a useful consequence. The keys and values of earlier tokens do not change when a new token is added. So the server computes them once, saves them, and reuses them at every later step. That saved set is the KV cache.
Without it, the model would recompute the keys and values for the whole prompt on every step. With it, each step computes them for the new token only and reads the rest from memory.
Why memory grows with context length
Every token adds its own keys and values, in every layer and every attention head. Nothing is thrown away while the conversation continues, so the cache grows in step with the number of tokens. The LowRAM authors state that growth as linear in context length.
It also multiplies across users. A server that handles many requests at once keeps a separate cache for each. The vLLM paper puts the consequence this way: high-throughput serving depends on batching many requests, but each request's cache is large and changes size as text is generated. When the cache fills the GPU memory, the server can batch fewer requests, and throughput falls.
This is the practical meaning of a long context window. The window sets how many tokens the model accepts. The cache is part of what it costs to hold them.
Paging: how vLLM wastes less memory
The first problem is not the size of the cache but how it is stored. The vLLM team, in Efficient Memory Management for Large Language Model Serving with PagedAttention, says poor memory management wastes space through fragmentation and redundant duplication, and that this caps the batch size.
Their fix borrows from operating systems. PagedAttention manages the cache the way virtual memory and paging manage a computer's RAM. The authors built the vLLM serving system on it and report near-zero waste in KV cache memory, plus flexible sharing of cache between requests. Against FasterTransformer and Orca, they report roughly 2 to 4 times the throughput at similar latency, with larger gains for longer sequences and bigger models. The code is public on GitHub.
Paging does not make the cache smaller. It stops the server from wasting memory around it.
Reusing the cache: prompt caching
If two requests start with the same text, the cache for that text is the same. An API can keep it and skip the work. Anthropic's prompt caching docs describe this as resuming processing from a previously cached prefix of the prompt. The docs say a cache hit needs identical content up to the marked point, that responses are the same as uncached ones, and that the default lifetime is 5 minutes, refreshed on each hit, with a 1-hour option.
On the price side, the docs list a read at 0.1 times the base input price for most models, with lower multipliers for some, and a 5-minute write at 1.25 times. Our explainer on what prompt caching is covers how to use it. Prompt caching saves compute and money across requests. It does not shrink the cache itself.
Making the cache smaller
The remaining approaches change what is stored. The LowRAM paper lists the families that earlier post-training methods use: exploiting redundancy across tokens, across layers and across feature dimensions, or lowering numerical precision. In short:
- Quantization stores keys and values with fewer bits.
- Eviction drops tokens that matter least.
- Low-rank compression stores a smaller approximation of the vectors, shared across layers.
What LowRAM adds
LowRAM, by Teresa Delgado and colleagues, was submitted on November 4, 2026. Its premise is that inside each attention head, values are largely predictable from their keys. It compresses keys with a cross-layer low-rank method called xKV. For values, it fits a small predictor per head at test time and stores corrections only for the tokens it gets most wrong. At each decoding step it rebuilds only a selected subset of the cache.
The authors test three models: Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct and Qwen3.5-4B. Scores are given as a percentage of the uncompressed baseline.
| Model | RULER 4K at 4x | LongBench at 4x |
|---|---|---|
| Mistral-7B | 96.0% | 97.9% |
| Llama-3.1-8B | 93.9% | 99.6% |
| Qwen3.5-4B | 100.1% | 94.4% |
At 4x compression, the paper reports up to about 5 times the baseline's peak decode throughput at 262K tokens of context, measured on Llama-3.1-8B. Combined with quantization and eviction, it reaches 26.9 times compression at 94.5% of baseline performance on Mistral with long LongBench contexts. Eviction alone reaches 10.0 times at 91.6%.
These are the authors' own numbers, from one paper posted a week ago. They also state limits: long generation was not evaluated, the predictor's fixed cost only pays off after a few thousand tokens, so short inputs such as RULER 4K lose more, and eviction combines less cleanly at high compression. We read the abstract and the results, and could not confirm from the text we read whether code is released.
What to watch
Compression, paging and caching attack different parts of the same bill: size, waste and repeat work. Providers rarely say which ones they use. The next thing to look for in a new model or serving system is a stated figure for memory per token of context, and whether long-context quality holds up at the compression it claims.
Frequently asked questions
What does KV stand for in KV cache?
Key and value. Each attention layer computes a key and a value vector for every token, and the cache stores them so they are not recomputed at each step.
Why does the KV cache use so much memory?
It holds keys and values for every token, in every layer and head, and a server keeps one cache per request. The LowRAM paper describes the growth as linear in context length.
Is a KV cache the same as prompt caching?
No. The KV cache works inside one generation. Prompt caching, as Anthropic's docs describe it, resumes processing from a cached prefix of the prompt across separate requests, which cuts cost and latency.
How can the KV cache be made smaller?
By storing fewer bits (quantization), dropping less useful tokens (eviction), or compressing the vectors across tokens, layers or feature dimensions. LowRAM combines several of these.

