ToolSite GGUF quantization explained
GGUF files for llama.cpp, Ollama and LM Studio come in a dozen quantization types. The name says how many bits each weight gets, which decides the file size, the memory the model needs and part of its quality. Llama 3.1 8B is 15.0 GB in BF16, 7.95 GB at Q8_0 and 4.52 GB at Q4_K_M.
How to read the names
- The number is the bits most weights are stored in: Q8 is 8-bit, Q4 is 4-bit, Q2 is 2-bit.
- _0 and _1 are the original formats: blocks of 32 weights share one scale (and, for _1, an offset). Q8_0 spends 8.5 bits per weight: 8 for the weight and a 16-bit scale per block of 32.
- _K marks the k-quants, which group weights into blocks of 256 and quantize the scales too, so they lose less quality for the same size.
- _S, _M and _L (small, medium, large) are mixes: the tensors that matter most, such as parts of attention and the feed-forward output, get a higher-bit type. Q4_K_M keeps some of them in Q6_K, which is why it averages about 4.8 bits rather than 4.5.
- IQ types (IQ4_XS, IQ3_XXS, IQ2_XS…) use an importance matrix measured on sample text to decide where precision goes. They reach smaller sizes at similar quality, but some run slower on CPUs and older GPUs.
Bits per weight and file size
Average bits per weight across a whole model, as llama.cpp reports them, and the resulting weights for a few popular models. The KV cache and runtime overhead come on top; the calculator adds them.
| Type | Bits/weight | Llama 3.1 8B | Gemma 4 12B | Qwen3.6 27B | Gemma 4 31B | Qwen3.6 35B-A3B | Llama 3.1 70B |
|---|---|---|---|---|---|---|---|
| BF16 (unquantized) | 16 | 15.0 GB | 22.3 GB | 51.7 GB | 58.3 GB | 67.0 GB | 131 GB |
| Q8_0 | 8.5 | 7.95 GB | 11.8 GB | 27.5 GB | 30.9 GB | 35.6 GB | 69.8 GB |
| Q6_K | 6.56 | 6.13 GB | 9.13 GB | 21.2 GB | 23.9 GB | 27.5 GB | 53.9 GB |
| Q5_K_M | 5.67 | 5.30 GB | 7.89 GB | 18.3 GB | 20.6 GB | 23.7 GB | 46.6 GB |
| Q4_K_M | 4.84 | 4.52 GB | 6.74 GB | 15.7 GB | 17.6 GB | 20.3 GB | 39.8 GB |
| Q3_K_M | 3.91 | 3.66 GB | 5.44 GB | 12.6 GB | 14.2 GB | 16.4 GB | 32.1 GB |
| Q2_K | 3.35 | 3.13 GB | 4.66 GB | 10.8 GB | 12.2 GB | 14.0 GB | 27.5 GB |
Which one to pick
- Q4_K_M is the usual default: about a third of the BF16 size, and for most models the drop in quality is hard to notice in everyday use.
- Q5_K_M and Q6_K are worth the extra memory for coding and other exact work, and for small models, which lose more from quantization than big ones.
- Q8_0 is close to the original; pick it when memory is not the constraint.
- Q3_K_M and Q2_K let a model fit where it otherwise would not, at a clear cost in quality. A big model at Q3 often still beats a smaller one at Q8, but check it on your own tasks.
- Leave room for context. The KV cache grows with every token: a type that only just fits leaves little space for a long conversation.
What fits on your GPU
Each page lists the most precise type every model fits in, with the longest context and the speed:
Model sizes computed from the files on Hugging Face on .