KV cache explained

The formula

For a model with plain grouped-query attention:

KV cache = 2 × layers × KV heads × head size × bytes per value × tokens × requests

The 2 counts keys and values. Llama 3.1 70B has 80 layers with 8 KV heads of 128 values, so each token costs 2 × 80 × 8 × 128 × 2 bytes = 320 KB in FP16, and a 128K-token conversation 40.0 GB. Several requests at once multiply it: an engine serving 8 chats of 32K tokens holds 8 × 10.0 GB.

Why some models need far less

• Latent attention (MLA) caches one compressed vector per token and layer instead of full keys and values. DeepSeek V3 stores 576 values per layer, 69 KB per token for a 671B model.
• Sliding windows keep only the most recent tokens in most layers, from 128 in gpt-oss to a few thousand in others. Gemma 4 31B: 10 of its 60 layers use full attention and 50 keep a sliding window of 1,024 tokens, so it needs 40 KB per token once the windows are full.
• Linear attention and state-space layers keep a fixed-size state instead of a cache. Qwen3.6 35B-A3B: 10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache, so only a quarter of it grows: 20 KB per token.
• A lower-precision cache halves or quarters the size: FP8 or Q8_0 cache in vLLM and llama.cpp takes half of FP16, Q4_0 a bit over a quarter, with a small loss of quality.

KV cache of popular models

FP16 cache for one request, for every tracked model with a context window of at least 128K tokens. Cheapest first.

DeepSeek V4 also shares and compresses its cache across layers, which is not modelled here, so its figures are upper bounds. Engines add their own margins: llama.cpp reserves the cache for the full context when it starts, and vLLM fills whatever memory the weights leave.