GGUF quantization explained
The best quant for a 16 GB card is the most precise one that still fits with room for context: on the RTX 4060 Ti 16GB at 8K context that is Q8_0 for Llama 3.1 8B (10.3 GB), Q8_0 for Gemma 4 12B (14.2 GB) and Q3_K_M for Qwen3.8 27B (15.0 GB). GGUF files for llama.cpp, Ollama and LM Studio come in a dozen quantization types. The name says how many bits each weight gets, which decides the file size, the memory the model needs and part of its quality. Llama 3.1 8B is 15.0 GB in BF16, 7.95 GB at Q8_0 and 4.52 GB at Q4_K_M.
How to read the names
- The number is the bits most weights are stored in: Q8 is 8-bit, Q4 is 4-bit, Q2 is 2-bit.
- _0 and _1 are the original formats: blocks of 32 weights share one scale (and, for _1, an offset). Q8_0 spends 8.5 bits per weight: 8 for the weight and a 16-bit scale per block of 32.
- _K marks the k-quants, which group weights into blocks of 256 and quantize the scales too, so they lose less quality for the same size.
- _S, _M and _L (small, medium, large) are mixes: the tensors that matter most, such as parts of attention and the feed-forward output, get a higher-bit type. Q4_K_M keeps some of them in Q6_K, which is why it averages about 4.8 bits rather than 4.5.
- IQ types (IQ4_XS, IQ3_XXS, IQ2_XS…) differ in their importance-matrix requirements. IQ4_XS is a non-linear 4-bit format with a nominal cost of 4.25 bits per weight; an importance matrix (imatrix) from sample text can improve quantization, but IQ4_XS does not require one. The name alone does not establish whether an imatrix was used or guarantee quality or speed: compare the specific model files and hardware backend.
Bits per weight and file size
Average bits per weight across a whole model, as llama.cpp reports them, and the resulting weights for a few popular models. The KV cache and runtime overhead come on top; the calculator adds them.
The table's 4.35 bits/weight for IQ4_XS is a whole-model estimate from public uploads, not its nominal 4.25. GGUF files mix tensor types; the IQ4_XS and IQ3_XXS estimates are medians of public uploads, whose tensors can use different precision. Two IQ4_XS files of the same model can differ by 5–10% (Qwen3.8 27B: 13.3 GB from unsloth, 14.3 GB from mradermacher). On a card where a gigabyte decides, go by the download size.
| Type | Bits/weight | Llama 3.1 8B | Gemma 4 12B | Qwen3.6 27B | Gemma 4 31B | Qwen3.6 35B-A3B | Llama 3.1 70B |
|---|---|---|---|---|---|---|---|
| BF16 (unquantized) | 16 | 15.0 GB | 22.3 GB | 51.7 GB | 58.3 GB | 67.0 GB | 131 GB |
| Q8_0 | 8.5 | 7.95 GB | 11.8 GB | 27.5 GB | 30.9 GB | 35.6 GB | 69.8 GB |
| Q6_K | 6.56 | 6.13 GB | 9.13 GB | 21.2 GB | 23.9 GB | 27.5 GB | 53.9 GB |
| Q5_K_M | 5.67 | 5.30 GB | 7.89 GB | 18.3 GB | 20.6 GB | 23.7 GB | 46.6 GB |
| Q4_K_M | 4.84 | 4.52 GB | 6.74 GB | 15.7 GB | 17.6 GB | 20.3 GB | 39.8 GB |
| IQ4_XS | 4.35 | 4.07 GB | 6.06 GB | 14.1 GB | 15.8 GB | 18.2 GB | 35.7 GB |
| Q3_K_M | 3.91 | 3.66 GB | 5.44 GB | 12.6 GB | 14.2 GB | 16.4 GB | 32.1 GB |
| IQ3_XXS | 3.3 | 3.08 GB | 4.59 GB | 10.7 GB | 12.0 GB | 13.8 GB | 27.1 GB |
| Q2_K | 3.35 | 3.13 GB | 4.66 GB | 10.8 GB | 12.2 GB | 14.0 GB | 27.5 GB |
Which one to pick
- Q4_K_M is the usual default: about a third of the BF16 size, and for most models the drop in quality is hard to notice in everyday use.
- Q5_K_M and Q6_K are worth the extra memory for coding and other exact work, and for small models, which lose more from quantization than big ones.
- Q8_0 is close to the original; pick it when memory is not the constraint.
- Q3_K_M and Q2_K let a model fit where it otherwise would not, at a clear cost in quality. A big model at Q3 often still beats a smaller one at Q8, but check it on your own tasks.
- Leave room for context. The KV cache grows with every token: a type that only just fits leaves little space for a long conversation.
What fits on your GPU
Each page lists the most precise type every model fits in, with the longest context and the speed:
Model sizes computed from the files on Hugging Face on .