KV cache quantization: q8_0 vs q4_0 vs f16, how much VRAM it saves and what it costs

Bytes per value

llama.cpp's cache types are its block formats: 32 values share one 16-bit scale (and, for the _1 types, a 16-bit minimum). Bits per value = block bytes × 8 ÷ 32. Read at llama.cpp 19e28a2 (ggml/src/ggml-common.h).

vLLM's FP8 cache is exactly 8 bits per value (50% of BF16); its scales are per tensor or per head, not per value.

KV cache of five models at f16, q8_0 and q4_0

One request, the site's own cache formula (the same as the calculator's KV cache setting). Gemma 4 and gpt-oss keep most layers at a sliding window, so their per-token figure is what each token adds once the windows are full. The “q8_0 K + q4_0 V” column is the mixed setting the quality test below favours (6.5 bits on average). GB means GiB.

Whole-model total at Q4_K_M weights with 32K context, with the site's 0.5 GB + 10% overhead; each figure opens the calculator with that cache type:

Where it matters most is context on a fixed card. On a 24 GB GPU at Q4_K_M, with 0.5 GB left free, the longest context goes for Qwen3.8 27B from 83K tokens at f16 to 158K at q8_0, and for Gemma 4 31B from 26K to 64K (134K at q4_0). A smaller cache matters less for gpt-oss-120b, whose 128-token windows keep the cache small next to its weights.

Checked against a llama.cpp log: in issue #22748 (2026-05-06; Gemma 4 31B Q5_K_M, -c 128000 -ctk q8_0 -ctv q4_0 -ub 256, on an RTX 5060 Ti + RTX 3060) the full-attention cache is 4062.50 MiB and the sliding-window cache 406.25 MiB across the two GPUs; at 6.5 bits per value the formula gives 4062.50 and 406.25 MiB (1280 sliding cells, because that run set -ub 256).

What it costs in quality

The one public side-by-side found is a perplexity and KL-divergence table posted in llama.cpp PR #7412 (2024-05-20), a research build that was not merged; the comment does not name the model or text. Rows are weights / K / V:

The author's reading: “The K cache seems to be much more sensitive to quantization than the V cache.” and “There seems to be no significant quality loss from using q8_0 instead of FP16 for the KV cache.” A q4_0 cache cost slightly more than Q4_K_M weights did in the same test, and a q8_0 K cache with a q4_0 V cache kept most of the q4_0 saving at a fraction of the loss. No public comparison was found for Qwen3.8 27B, Gemma 4, gpt-oss or Llama 3.1, or for vLLM's FP8 cache on them, so treat these as one data point rather than a rule.