How much VRAM do I need for an LLM?

About 18.3 GB with 8,192 tokens of context and 19.9 GB with 32,768, for one request with an FP16 KV cache; the weights alone take 15.7 GB. So it fits one 24 GB card such as an RTX 3090 or 4090 with a 32K context.

Weights = parameters × bits per weight ÷ 8. KV cache = 2 × layers × KV heads × head dimension × bytes per value × tokens × requests. Runtime overhead = 0.5 GB for the CUDA context plus 10% of weights and cache for activations and fragmentation; you can change the 10% under Advanced.

The KV cache grows with every token of every request. Llama 3.1 8B needs 128 KB of FP16 cache per token, so one 128K-token conversation adds 16 GB on top of the weights. An FP8 cache halves that.

They compute with a few experts per token, but every expert has to be loaded, so weight memory follows the total parameter count: 30.5 billion for Qwen3-30B-A3B, not 3 billion.

Q4_K_M is the usual default at about a third of the BF16 size; Q5_K_M and Q6_K keep more quality for coding and small models; Q3_K_M and Q2_K make a model fit at a clear cost in quality. The GGUF quantization guide on this site lists the bits per weight of each type and the size of popular models.

No. Weight precision and KV cache precision are separate settings, so Q4 weights with an FP16 cache still need the full-size cache.

Architecture data comes from model configs or GGUF headers, with runtime details checked against inference-engine sources. Multi-head latent attention (DeepSeek V3, GLM-5.3, Kimi K3) caches 576 values per token per layer instead of full keys and values; DeepSeek V3.2 and ordinary GLM-5 indexers add a 128-value FP8 key per token; GLM-5.3-Flash instead adds three 128-value vectors at the selected KV precision, following llama.cpp 5fc4f3c. Sliding-window layers keep only their window: 128 tokens in half of gpt-oss, 1,024 tokens in 50 of Gemma 4 31B’s 60 layers, whose other 10 layers cache 4 heads of 512 values that serve as both keys and values. Linear-attention layers keep a fixed-size state instead of a growing cache: 48 of Qwen3.8 27B’s 64 layers and 69 of Kimi K3’s 93. DeepSeek V4 also shares and compresses its cache across layers, which the calculator does not model, so its KV figure is an upper bound.

It uses the size of the model’s weight files on Hugging Face, so it matches the download even for checkpoints that mix formats, such as MXFP4 experts in gpt-oss or the 4-bit experts of DeepSeek V4 and MiMo V2.6.

It is an estimate. Inference engines add their own buffers: vLLM reserves a fixed share of GPU memory up front, and llama.cpp allocates the cache for the full context when it starts. Leave some headroom, especially on a GPU that also drives your display.

Since b7440, llama.cpp fits unset arguments to free device memory (-fit, on by default). It keeps a -c you set and chooses how many layers go on the GPU, but a number given to -ngl stops the fit, and llama.cpp then loads exactly that many layers, which can run out of memory on a smaller card. So the commands set -c and leave -ngl at auto. On old builds such as b6000, which put no layer on the GPU by default, add -ngl 99. CPU-only commands keep -ngl 0 so that no layer goes to a GPU.

The calculation runs in your browser. The only network requests go to Hugging Face, when you load a model.