LLM VRAM Calculator

Weights = parameters × bits per weight ÷ 8. KV cache = 2 × layers × KV heads × head dimension × bytes per value × tokens × requests. Runtime overhead = 0.5 GB for the CUDA context plus 10% of weights and cache for activations and fragmentation; you can change the 10% under Advanced.

The KV cache grows with every token of every request. Llama 3.1 8B needs 128 KB of FP16 cache per token, so one 128K-token conversation adds 16 GB on top of the weights. An FP8 cache halves that.

They compute with a few experts per token, but every expert has to be loaded, so weight memory follows the total parameter count: 30.5 billion for Qwen3-30B-A3B, not 3 billion.

Q4_K_M is the usual default at about a third of the BF16 size; Q5_K_M and Q6_K keep more quality for coding and small models; Q3_K_M and Q2_K make a model fit at a clear cost in quality. The GGUF quantization guide on this site lists the bits per weight of each type and the size of popular models.

No. Weight precision and KV cache precision are separate settings, so Q4 weights with an FP16 cache still need the full-size cache.

Everything comes from the model’s config.json. Multi-head latent attention (DeepSeek V3, GLM-5.3, Kimi K3) caches 576 values per token per layer instead of full keys and values; DeepSeek V3.2 and GLM-5 add a 128-value FP8 key per token for their sparse-attention indexer. Sliding-window layers keep only their window: 128 tokens in half of gpt-oss, 1,024 tokens in 50 of Gemma 4 31B’s 60 layers, whose other 10 layers cache 4 heads of 512 values that serve as both keys and values. Linear-attention layers keep a fixed-size state instead of a growing cache: 48 of Qwen3.8 27B’s 64 layers and 69 of Kimi K3’s 93. DeepSeek V4 also shares and compresses its cache across layers, which the calculator does not model, so its KV figure is an upper bound.

It uses the size of the model’s weight files on Hugging Face, so it matches the download even for checkpoints that mix formats, such as MXFP4 experts in gpt-oss or the 4-bit experts of DeepSeek V4 and MiMo V2.6.

It is an estimate. Inference engines add their own buffers: vLLM reserves a fixed share of GPU memory up front, and llama.cpp allocates the cache for the full context when it starts. Leave some headroom, especially on a GPU that also drives your display.

The calculation runs in your browser. The only network requests go to Hugging Face, when you load a model.