ToolSite Which GPUs can run popular LLMs?
A reproducible comparison of 41 public models at Q4_K_M, FP8 and BF16. Every row links to its model-specific calculations. Download the raw CSV.
What fits on one GPU?
Each count uses 8,192 context tokens, one request, an FP16 KV cache, and the runtime overhead described below. These are memory-capacity comparisons, not measured deployment results.
| GPU memory | Q4_K_M | FP8 | BF16 |
|---|---|---|---|
| 24 GiB | 18 / 41 | 8 / 41 | 6 / 41 |
| 48 GiB | 19 / 41 | 18 / 41 | 8 / 41 |
| 80 GiB | 24 / 41 | 19 / 41 | 18 / 41 |
For a specific RTX 4090, L40S, H100 or multi-GPU setup, open a model row or use the interactive calculator.
All model estimates
Totals are in GiB and include weights, KV cache and runtime overhead. “Yes” means the Q4_K_M estimate fits on one GPU of the listed capacity.
| Model | Attention layout | Q4_K_M | FP8 | BF16 | 24 GiB | 48 GiB | 80 GiB |
|---|---|---|---|---|---|---|---|
| AliceAI Foundation 80B-A3B (MoE)The 36 KDA layers keep a fixed recurrent state, which this estimate does not include; only the 12 gated-attention layers add a context-growing KV cache. Real serving memory may be higher. | 12 of its 48 layers use full attention and 36 are KDA recurrent layers with no growing cache | 51.09 | 83.98 | 167.25 | No | No | Yes |
| DeepSeek V4 Flash 0731This model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound. | All 43 layers use full attention | 189.77 | 312.86 | 624.48 | No | No | No |
| Qwen3.6 35B-A3B (MoE) | 10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache | 22.95 | 37.5 | 74.33 | Yes | Yes | Yes |
| Qwen3.6 27B | 16 of its 64 layers use full attention and 48 are linear-attention layers with no growing cache | 18.27 | 29.51 | 57.97 | Yes | Yes | Yes |
| Qwen3.8 27B | 16 of its 64 layers use full attention and 48 are linear-attention layers with no growing cache | 18.27 | 29.51 | 57.97 | Yes | Yes | Yes |
| Qwen3.8 Flash Next (180B MoE) | 12 of its 48 layers use full attention and 36 are linear-attention layers with no growing cache | 112.27 | 185.11 | 369.51 | No | No | No |
| Qwen3.8 2.4T-A95B (MoE) | 23 of its 92 layers use full attention and 69 are linear-attention layers with no growing cache | 1517.42 | 2507.29 | 5013.3 | No | No | No |
| Qwen3.5 9B | 8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache | 6.76 | 10.66 | 20.55 | Yes | Yes | Yes |
| Qwen3.5 122B-A10B (MoE) | 12 of its 48 layers use full attention and 36 are linear-attention layers with no growing cache | 78.23 | 128.85 | 257 | No | No | Yes |
| Qwen3-Coder-Next (80B MoE) | 12 of its 48 layers use full attention and 36 are linear-attention layers with no growing cache | 50.09 | 82.33 | 163.95 | No | No | Yes |
| DeepSeek V4.1 FlashThis model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound. | All 40 layers use full attention | 474.22 | 783.06 | 1564.93 | No | No | No |
| DeepSeek V4 FlashThis model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound. | All 43 layers use full attention | 181.57 | 299.3 | 597.36 | No | No | No |
| DeepSeek V4 ProThis model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound. | All 61 layers use full attention | 992.5 | 1639.49 | 3277.43 | No | No | No |
| DeepSeek V3.2 | All 61 layers use multi-head latent attention | 425.94 | 703.27 | 1405.39 | No | No | No |
| GLM-5.3 | All 78 layers use multi-head latent attention | 468.19 | 773.03 | 1544.78 | No | No | No |
| GLM-5.3 Flash | 11 of its 45 layers use multi-head latent attention and 34 are linear-attention layers with no growing cache | 199.76 | 329.79 | 658.97 | No | No | No |
| GLM-5.2 | All 78 layers use multi-head latent attention | 468.19 | 773.03 | 1544.78 | No | No | No |
| GLM-4.7 Flash | All 47 layers use multi-head latent attention | 20.31 | 32.94 | 64.92 | Yes | Yes | Yes |
| Gemma 4 31B | 10 of its 60 layers use full attention and 50 keep a sliding window of 1,024 tokens | 21.09 | 33.74 | 65.78 | Yes | Yes | Yes |
| Gemma 4 26B-A4B (MoE) | 5 of its 30 layers use full attention and 25 keep a sliding window of 1,024 tokens | 16.8 | 27.24 | 53.67 | Yes | Yes | Yes |
| Gemma 4 12B | 8 of its 48 layers use full attention and 40 keep a sliding window of 1,024 tokens | 8.33 | 13.16 | 25.42 | Yes | Yes | Yes |
| Gemma 4 E4B | 4 of its 42 layers use full attention, 20 keep a sliding window of 512 tokens and 18 reuse the cache of earlier layers | 5.61 | 8.85 | 17.04 | Yes | Yes | Yes |
| Kimi K3 | 24 of its 93 layers use multi-head latent attention and 69 are linear-attention layers with no growing cache | 1723.72 | 2848.65 | 5696.56 | No | No | No |
| MiniMax M3 | All 60 layers use full attention | 266.21 | 439.01 | 876.5 | No | No | No |
| MiniMax M2.7 | All 62 layers use full attention | 144.37 | 236.91 | 471.2 | No | No | No |
| MiMo V2.6 Flash | 9 of its 48 layers use full attention and 39 keep a sliding window of 128 tokens | 193.32 | 319.08 | 637.43 | No | No | No |
| MiMo V2.6 Pro | 10 of its 70 layers use full attention and 60 keep a sliding window of 128 tokens | 635.77 | 1050.23 | 2099.5 | No | No | No |
| Mistral Medium 3.5 128B | All 88 layers use full attention | 82.68 | 134.35 | 265.18 | No | No | No |
| Nemotron 3 Nano 4B | 4 of its 42 layers use full attention and 38 are Mamba or feed-forward layers with no growing cache | 3.1 | 4.71 | 8.78 | Yes | Yes | Yes |
| Nemotron 3 Nano 30B-A3B (MoE) | 6 of its 52 layers use full attention and 46 are Mamba or feed-forward layers with no growing cache | 20.12 | 32.9 | 65.25 | Yes | Yes | Yes |
| Nemotron 3 Super 120B-A12B (MoE) | 8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache | 77.18 | 127.2 | 253.84 | No | No | Yes |
| Xing 4.0 29B-A4B (MoE) | All 40 layers use multi-head latent attention | 20.23 | 32.87 | 64.84 | Yes | Yes | Yes |
| Muse Glimmer 30B | 13 of its 52 layers use full attention and 39 keep a sliding window of 2,048 tokens | 19.15 | 31.2 | 61.71 | Yes | Yes | Yes |
| MiniCPM5 2B | All 42 layers use full attention | 2.42 | 3.44 | 6.02 | Yes | Yes | Yes |
| Llama 3.1 8B | All 32 layers use full attention | 6.58 | 9.83 | 18.05 | Yes | Yes | Yes |
| Llama 3.1 70B | All 80 layers use full attention | 46.98 | 75.53 | 147.81 | No | Yes | Yes |
| Qwen3 8B | All 36 layers use full attention | 6.81 | 10.13 | 18.52 | Yes | Yes | Yes |
| Qwen3 30B-A3B (MoE) | All 48 layers use full attention | 20.25 | 32.6 | 63.88 | Yes | Yes | Yes |
| gpt-oss-20b | 12 of its 24 layers use full attention and 12 keep a sliding window of 128 tokens | 13.67 | 22.14 | 43.56 | Yes | Yes | Yes |
| gpt-oss-120b | 18 of its 36 layers use full attention and 18 keep a sliding window of 128 tokens | 73.22 | 120.5 | 240.19 | No | No | Yes |
| DeepSeek V3 / R1 (671B) | All 61 layers use multi-head latent attention | 425.36 | 702.36 | 1403.63 | No | No | No |
How to reproduce the estimates
Q4_K_M uses an average 4.84 bits per parameter, including GGUF quantization scales. FP8 uses 8 bits and BF16 uses 16 bits. Weight memory is total parameters × bits per parameter ÷ 8; for mixture-of-experts models, all experts count because they must be loaded even though only some run for each token.
The FP16 KV cache is calculated from each model's attention layout: full attention grows with context; sliding-window layers keep only their window; MLA stores a compressed latent; linear or state-space layers do not add a context-growing KV cache. The total is weights + cache + 0.5 GiB + 10% of weights and cache. See the KV-cache explanation and calculation source and tests.
The CSV records the weight, cache and overhead components, the three total precisions, model ID, source-check date, architecture summary and per-model URL. Model files were checked on the dates in the CSV; this report was updated .
Limits
These are estimates, not measured peak GPU usage. vLLM may reserve more memory and llama.cpp can allocate the full configured cache at startup. CUDA graphs, activation buffers, vision encoders and host-side offload can change the real requirement. A “Yes” near a GPU's limit is not a deployment guarantee. Models with a known unmodeled cache optimization carry a note in the table and on their own page.
Reuse the CSV with attribution to this report or the MIT-licensed source. There is no requirement to link back.