Which GPUs can run popular LLMs?

A reproducible comparison of 69 public models at Q4_K_M, FP8 and BF16. Every row links to its model-specific calculations. Download the raw CSV.

What fits on one GPU?

Each count uses 8,192 context tokens, one request, an FP16 KV cache, and the runtime overhead described below. These are memory-capacity comparisons, not measured deployment results.

GPU memory	Q4_K_M	FP8	BF16
24 GB	38 / 69	21 / 69	19 / 69
48 GB	40 / 69	39 / 69	21 / 69
80 GB	47 / 69	40 / 69	39 / 69

GB = GiB (1024³ bytes), as everywhere on this site; the CSV columns ending in _gib use the same unit.

For a specific RTX 4090, L40S, H100 or multi-GPU setup, open a model row or use the interactive calculator.

For 32K context on one 8 to 96 GB card, see the models that fit each VRAM size. The same numbers, at 8K, 32K and 128K, are in the open dataset (CSV and JSON, CC BY 4.0).

All model estimates

Totals are in GB and include weights, KV cache and runtime overhead. “Yes” means the Q4_K_M estimate fits on one GPU of the listed capacity.

Model	Q4_K_M	FP8	BF16	24 GB	48 GB	80 GB	Attention layout
DeepSeek V4.1 Flash 1017 (MoE)This model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound.	474.22	783.06	1564.93	No	No	No	The tables use an independent-layer upper-bound proxy for its 40-layer shared, compressed cache
Nemotron 3.5 Super VL 120B-A12B (MoE)Only 8 attention blocks keep a context-growing KV cache; the other 80 blocks are 40 Mamba-2 and 40 MoE blocks. Fixed Mamba state, vision activations, image/video tokens and MTP need additional runtime memory. The 1,048,576-token context is shared by text, visual inputs and generated output. Quantized rows are storage estimates, not verified runtime support.	77.7	128.07	255.56	No	No	Yes	8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache
Ling 3.1 Flash 560B-A25B (MoE)Only the 9 MLA layers keep a context-growing KV cache (576 values per token per layer). The 63 KDA layers keep fixed recurrent state, left to the runtime allowance. The model card advertises 1M tokens, but config.json sets 262,144; the tables use the configured limit. MTP and serving buffers may need more memory.	355.18	586.7	1172.81	No	No	No	9 of its 72 layers use multi-head latent attention and 63 are KDA recurrent layers with no growing cache
LightOnOCR-3 4BOnly the 8 full-attention layers keep a KV cache. The 24 linear-attention layers hold a fixed state per request (48 MB in FP32), left to the overhead allowance, as is the vision encoder’s working memory while it reads a page. A page takes about 1,800 to 4,800 image tokens depending on its render size, plus its output: 1,158 tokens on average for plain transcription, 1,433 with grounding.	3.59	5.43	10.08	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
Kolibri-1 (MoE)Kolibri-1 is served with Aleph Alpha’s vLLM plugin. The tables use this site’s FP16 cache and runtime allowance; the publisher evaluates an FP8 cache and recommends up to 262,144 tokens for serving. Its validated 1,048,576-token setting needs explicit overrides. GGUF sizes here are estimates, not verified compatible downloads.	49.19	80.79	160.81	No	No	Yes	10 of its 50 layers use full attention and 40 keep a sliding window of 513 tokens
ThinkingCap Qwen3.8 27B	18.27	29.51	57.97	Yes	Yes	Yes	16 of its 64 layers use full attention and 48 are linear-attention layers with no growing cache
Bespoke Nimble 9B	6.32	9.95	19.12	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
LensVLM 9B	6.61	10.41	20.05	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
Qwen3-Coder 30B-A3B (MoE)	20.25	32.6	63.88	Yes	Yes	Yes	All 48 layers use full attention
Ornith 1.0 35B (MoE)	22.43	36.64	72.6	Yes	Yes	Yes	10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache
Ornith 1.0 9B	6.61	10.41	20.05	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
Granite 4.2 30B	20.85	32.69	62.69	Yes	Yes	Yes	All 64 layers use full attention
Granite 4.2 8B	7.32	10.88	19.89	Yes	Yes	Yes	All 40 layers use full attention
Granite 4.2 3B	3.46	4.94	8.69	Yes	Yes	Yes	All 40 layers use full attention
Spark-X2.5 4B	3.47	5.14	9.35	Yes	Yes	Yes	9 of its 36 layers use full attention and 27 keep a sliding window of 512 tokens
Ling 3.0 Tiny 7.9B-A1.3B (MoE)The 18 KDA layers keep a fixed recurrent state, which this estimate does not include; only the 6 MLA layers add a context-growing KV cache. Real serving memory may be higher.	5.45	8.64	16.73	Yes	Yes	Yes	6 of its 24 layers use multi-head latent attention and 18 are KDA recurrent layers with no growing cache
LFM2.5 8B-A1B (MoE)	5.85	9.28	17.95	Yes	Yes	Yes	6 of its 24 layers use full attention and 18 are short-convolution layers with no growing cache
Step 3.7 Flash 196B-A11B (MoE)	125.86	207.34	413.63	No	No	No	12 of its 45 layers use full attention and 33 keep a sliding window of 512 tokens
Hy4 Preview 770B-A49B (MoE)	484.69	800.31	1599.35	No	No	No	All 78 layers use multi-head latent attention
Limite 1B Violetto	1.31	1.73	2.79	Yes	Yes	Yes	12 of its 48 layers use full attention and 36 keep a sliding window of 1,025 tokens
ZDTaichu 5.0 9B	6.85	10.81	20.84	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
Hemmingway-1 27B	17.98	29.04	57.03	Yes	Yes	Yes	16 of its 64 layers use full attention and 48 are linear-attention layers with no growing cache
MiMo V2.6 Distill Qwen 9B	6.61	10.41	20.05	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
K2-Horizon MoVA 36B-A4B (MoE)	25.36	40.51	78.87	No	Yes	Yes	All 48 layers use full attention
IQuest-Q1 320B-A15B (MoE)	201.11	330.73	658.88	No	No	No	25 of its 88 layers use full attention and 63 keep a sliding window of 4,096 tokens
LLM-jp-4.1 32B-A3B Thinking (MoE)	20.97	33.97	66.9	Yes	Yes	Yes	All 32 layers use full attention
Ornith 1.5 35B-A3B (MoE)	22.95	37.5	74.33	Yes	Yes	Yes	10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache
Ornith 1.5 9B	6.76	10.66	20.55	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
AliceAI Foundation 80B-A3B (MoE)The 36 KDA layers keep a fixed recurrent state, which this estimate does not include; only the 12 gated-attention layers add a context-growing KV cache. Real serving memory may be higher.	51.09	83.98	167.25	No	No	Yes	12 of its 48 layers use full attention and 36 are KDA recurrent layers with no growing cache
DeepSeek V4 Flash 0731This model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound.	189.77	312.86	624.48	No	No	No	The tables use an independent-layer upper-bound proxy for its 43-layer shared, compressed cache
Qwen3.6 35B-A3B (MoE)	22.95	37.5	74.33	Yes	Yes	Yes	10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache
Qwen3.6 27B	18.27	29.51	57.97	Yes	Yes	Yes	16 of its 64 layers use full attention and 48 are linear-attention layers with no growing cache
Qwen3.8 27B	18.27	29.51	57.97	Yes	Yes	Yes	16 of its 64 layers use full attention and 48 are linear-attention layers with no growing cache
Qwen3.8 Flash Next (180B MoE)	112.27	185.11	369.51	No	No	No	12 of its 48 layers use full attention and 36 are linear-attention layers with no growing cache
Qwen3.8 2.4T-A95B (MoE)	1517.42	2507.29	5013.3	No	No	No	23 of its 92 layers use full attention and 69 are linear-attention layers with no growing cache
Qwen3.5 9B	6.76	10.66	20.55	Yes	Yes	Yes	8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache
Qwen3.5 122B-A10B (MoE)	78.23	128.85	257	No	No	Yes	12 of its 48 layers use full attention and 36 are linear-attention layers with no growing cache
Qwen3-Coder-Next (80B MoE)	50.09	82.33	163.95	No	No	Yes	12 of its 48 layers use full attention and 36 are linear-attention layers with no growing cache
DeepSeek V4.1 FlashThis model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound.	474.22	783.06	1564.93	No	No	No	The tables use an independent-layer upper-bound proxy for its 40-layer shared, compressed cache
DeepSeek V4 FlashThis model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound.	181.57	299.3	597.36	No	No	No	The tables use an independent-layer upper-bound proxy for its 43-layer shared, compressed cache
DeepSeek V4 ProThis model shares and compresses its KV cache across layers, which the calculator does not model, so the KV cache figure is an upper bound.	992.5	1639.49	3277.43	No	No	No	The tables use an independent-layer upper-bound proxy for its 61-layer shared, compressed cache
DeepSeek V3.2	425.94	703.27	1405.39	No	No	No	All 61 layers use multi-head latent attention
GLM-5.3	468.19	773.03	1544.78	No	No	No	All 78 layers use multi-head latent attention
GLM-5.3 FlashGLM-5-Next cache uses llama.cpp 5fc4f3c: the indexer stores key, gate and pooled values at the selected KV precision for every context row. Fixed recurrent state, MTP and scratch buffers are excluded. Quantized sizes do not establish kernel support or measured GPU memory.	199.82	329.85	659.03	No	No	No	11 of its 45 layers use multi-head latent attention and 34 are linear-attention layers with no growing cache
GLM-5.2	468.19	773.03	1544.78	No	No	No	All 78 layers use multi-head latent attention
GLM-4.7 Flash	20.31	32.94	64.92	Yes	Yes	Yes	All 47 layers use multi-head latent attention
Gemma 4 31B	21.86	34.51	66.55	Yes	Yes	Yes	10 of its 60 layers use full attention and 50 keep a sliding window of 1,024 tokens
Gemma 4 26B-A4B (MoE)	16.99	27.43	53.87	Yes	Yes	Yes	5 of its 30 layers use full attention and 25 keep a sliding window of 1,024 tokens
Gemma 4 12B	8.57	13.41	25.66	Yes	Yes	Yes	8 of its 48 layers use full attention and 40 keep a sliding window of 1,024 tokens
Gemma 4 E4B	5.64	8.87	17.06	Yes	Yes	Yes	4 of its 42 layers use full attention, 20 keep a sliding window of 512 tokens and 18 reuse the cache of earlier layers
Kimi K3	1723.72	2848.65	5696.56	No	No	No	24 of its 93 layers use multi-head latent attention and 69 are linear-attention layers with no growing cache
MiniMax M3	266.21	439.01	876.5	No	No	No	All 60 layers use full attention
MiniMax M2.7	144.37	236.91	471.2	No	No	No	All 62 layers use full attention
MiMo V2.6 Flash	193.46	319.21	637.56	No	No	No	9 of its 48 layers use full attention and 39 keep a sliding window of 128 tokens
MiMo V2.6 Pro	635.98	1050.44	2099.7	No	No	No	10 of its 70 layers use full attention and 60 keep a sliding window of 128 tokens
Mistral Medium 3.5 128B	82.68	134.35	265.18	No	No	No	All 88 layers use full attention
Nemotron 3 Nano 4B	3.1	4.71	8.78	Yes	Yes	Yes	4 of its 42 layers use full attention and 38 are Mamba or feed-forward layers with no growing cache
Nemotron 3 Nano 30B-A3B (MoE)	20.12	32.9	65.25	Yes	Yes	Yes	6 of its 52 layers use full attention and 46 are Mamba or feed-forward layers with no growing cache
Nemotron 3 Super 120B-A12B (MoE)	77.18	127.2	253.84	No	No	Yes	8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache
Xing 4.0 29B-A4B (MoE)	20.23	32.87	64.84	Yes	Yes	Yes	All 40 layers use multi-head latent attention
Muse Glimmer 30B	19.17	31.22	61.73	Yes	Yes	Yes	13 of its 52 layers use full attention and 39 keep a sliding window of 2,048 tokens
MiniCPM5 2B	2.42	3.44	6.02	Yes	Yes	Yes	All 42 layers use full attention
Llama 3.1 8B	6.58	9.83	18.05	Yes	Yes	Yes	All 32 layers use full attention
Llama 3.1 70B	46.98	75.53	147.81	No	Yes	Yes	All 80 layers use full attention
Qwen3 8B	6.81	10.13	18.52	Yes	Yes	Yes	All 36 layers use full attention
Qwen3 30B-A3B (MoE)	20.25	32.6	63.88	Yes	Yes	Yes	All 48 layers use full attention
gpt-oss-20b	13.69	22.15	43.58	Yes	Yes	Yes	12 of its 24 layers use full attention and 12 keep a sliding window of 128 tokens
gpt-oss-120b	73.25	120.52	240.21	No	No	Yes	18 of its 36 layers use full attention and 18 keep a sliding window of 128 tokens
DeepSeek V3 / R1 (671B)	425.36	702.36	1403.63	No	No	No	All 61 layers use multi-head latent attention

Not in the table until its weights are out: Mistral Large 4, announced on October 6, 2026 with 1.05T parameters, estimated from Mistral's figures until the release on October 31.

The real Q4_K_M files on Hugging Face

For 47 of the 69 models, the Q4_K_M GGUF llama.cpp downloads with -hf <repo>:Q4_K_M (MXFP4 for gpt-oss; 1 marked row is a community conversion that stock llama.cpp cannot load), with the bytes the Hugging Face file list reported on 2026-09-29. The estimates above assume 4.84 bits per parameter for Q4_K_M; across these 45 Q4_K_M files the median is 4.91 bits (range 4.54 to 5.84). 9 files are more than 5% larger than the estimate and 1 more than 5% smaller: Q4_K_M mixes Q4_K with Q6_K and other block types tensor by tensor, and uploaders differ in which tensors they keep at higher precision. With the file's own bytes, 31 of these fit one 24 GB card with an 8K context and 0.5 GB free.

Model	File on Hugging Face	Bytes	Bits per weight	vs estimate	Total, 8K	24 GB
ThinkingCap Qwen3.8 27B	bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF	17,442,400,352	5.02	+4%	18.9 GB	Yes
LensVLM 9B	bartowski/LensVLM-9B-GGUF	5,841,052,896	4.97	+3%	6.76 GB	Yes
Qwen3-Coder 30B-A3B (MoE)	unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF	18,556,689,568	4.86	±0%	20.3 GB	Yes
Ornith 1.0 35B (MoE)	bartowski/deepreinforce-ai_Ornith-1.0-35B-GGUF	21,391,448,128	4.87	+1%	22.6 GB	Yes
Ornith 1.0 9B	bartowski/deepreinforce-ai_Ornith-1.0-9B-GGUF	5,910,782,656	5.03	+4%	6.83 GB	Yes
Granite 4.2 30B	bartowski/granite-4.2-30b-GGUF	18,027,639,840	4.93	+2%	21.2 GB	Yes
Granite 4.2 8B	bartowski/granite-4.2-8b-GGUF	5,539,283,360	5.04	+4%	7.55 GB	Yes
Granite 4.2 3B	bartowski/granite-4.2-3b-GGUF	2,317,126,048	5.07	+5%	3.56 GB	Yes
Spark-X2.5 4B	XHToken/Spark-X2.5-4B-GGUF	2,600,224,352	5.06	+5%	3.59 GB	Yes
Ling 3.0 Tiny 7.9B-A1.3B (MoE)	bartowski/Ling-3.0-tiny-GGUF	4,917,354,688	4.98	+3%	5.60 GB	Yes
LFM2.5 8B-A1B (MoE)	unsloth/LFM2.5-8B-A1B-GGUF	5,322,223,200	5.03	+4%	6.06 GB	Yes
Step 3.7 Flash 196B-A11B (MoE)	unsloth/Step-3.7-Flash-GGUF4 parts	122,090,426,976	4.85	±0%	126 GB	No
Hemmingway-1 27B	bartowski/Altworld_Hemmingway-1-GGUF	17,442,400,320	5.11	+6%	18.9 GB	Yes
MiMo V2.6 Distill Qwen 9B	bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF	5,841,049,120	4.97	+3%	6.76 GB	Yes
K2-Horizon MoVA 36B-A4B (MoE)	IFM/K2-Horizon-MoVA-36B-A4B-GGUF	22,368,011,616	4.78	−1%	25.1 GB	No
Ornith 1.5 35B-A3B (MoE)	bartowski/Ornith-1.5-35B-A3B-GGUF	21,864,081,056	4.87	+1%	23.1 GB	Yes
Ornith 1.5 9B	bartowski/Ornith-1.5-9B-GGUF	5,910,782,848	4.90	+1%	6.83 GB	Yes
AliceAI Foundation 80B-A3B (MoE)community conversion, needs a patched llama.cpp	Yamada114514/AliceAI-Foundation-80B-A3B-Base-GGUF	48,421,862,304	4.77	−2%	50.3 GB	No
Qwen3.6 35B-A3B (MoE)	ggml-org/Qwen3.6-35B-A3B-GGUF	20,419,565,568	4.54	−6%	21.6 GB	Yes
Qwen3.6 27B	ggml-org/Qwen3.6-27B-GGUF	19,095,766,304	5.50	+14%	20.6 GB	Yes
Qwen3.8 27B	ggml-org/Qwen3.8-27B-GGUF	18,973,870,528	5.46	+13%	20.5 GB	Yes
Qwen3.8 Flash Next (180B MoE)	bartowski/Qwen3.8-Flash-Next-GGUF4 parts	119,599,030,080	5.32	+10%	123 GB	No
Qwen3.5 9B	unsloth/Qwen3.5-9B-GGUF	5,680,522,464	4.71	−3%	6.59 GB	Yes
Qwen3.5 122B-A10B (MoE)	unsloth/Qwen3.5-122B-A10B-GGUF3 parts	76,536,964,608	4.89	+1%	79.1 GB	No
Qwen3-Coder-Next (80B MoE)	unsloth/Qwen3-Coder-Next-GGUF	48,528,320,544	4.87	+1%	50.4 GB	No
DeepSeek V3.2	unsloth/DeepSeek-V3.2-GGUF9 parts	405,398,024,320	4.73	−2%	416 GB	No
GLM-5.2	unsloth/GLM-5.2-GGUF11 parts	465,825,525,088	4.95	+2%	478 GB	No
GLM-4.7 Flash	unsloth/GLM-4.7-Flash-GGUF	18,312,339,808	4.69	−3%	19.7 GB	Yes
Gemma 4 31B	unsloth/gemma-4-31B-it-GGUF	18,323,733,440	4.69	−3%	21.2 GB	Yes
Gemma 4 26B-A4B (MoE)	unsloth/gemma-4-26B-A4B-it-GGUF	16,947,541,728	5.25	+9%	18.4 GB	Yes
Gemma 4 12B	unsloth/gemma-4-12b-it-GGUF	7,121,861,440	4.76	−2%	8.45 GB	Yes
Gemma 4 E4B	unsloth/gemma-4-E4B-it-GGUF	4,977,171,584	4.98	+3%	5.78 GB	Yes
MiniMax M3	unsloth/MiniMax-M3-GGUF7 parts	264,031,796,736	4.95	+2%	272 GB	No
MiniMax M2.7	unsloth/MiniMax-M2.7-GGUF4 parts	140,169,905,504	4.90	+1%	146 GB	No
Mistral Medium 3.5 128B	unsloth/Mistral-Medium-3.5-128B-GGUF3 parts	74,897,139,136	4.69	−3%	80.3 GB	No
Nemotron 3 Nano 4B	unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF	2,900,295,712	5.84	+21%	3.61 GB	Yes
Nemotron 3 Nano 30B-A3B (MoE)	ggml-org/NVIDIA-Nemotron-3-Nano-30B-A3B-GGUF	22,421,827,488	5.68	+17%	23.5 GB	No
Nemotron 3 Super 120B-A12B (MoE)	unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF3 parts	82,541,168,480	5.34	+10%	85.1 GB	No
Muse Glimmer 30B	bartowski/Muse-Glimmer-30B-GGUF	17,306,324,000	4.65	−4%	18.4 GB	Yes
MiniCPM5 2B	bartowski/MiniCPM5-2B-GGUF	1,615,826,144	5.14	+6%	2.52 GB	Yes
Llama 3.1 8B	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	4,920,739,232	4.90	+1%	6.64 GB	Yes
Llama 3.1 70B	bartowski/Meta-Llama-3.1-70B-Instruct-GGUF	42,520,398,400	4.82	±0%	46.8 GB	No
Qwen3 8B	unsloth/Qwen3-8B-GGUF	5,027,784,512	4.91	+1%	6.89 GB	Yes
Qwen3 30B-A3B (MoE)	ggml-org/Qwen3-30B-A3B-GGUF	18,556,686,240	4.86	±0%	20.3 GB	Yes
gpt-oss-20bMXFP4	ggml-org/gpt-oss-20b-GGUF	12,109,566,624	4.63	−12%	13.1 GB	Yes
gpt-oss-120bMXFP4	ggml-org/gpt-oss-120b-GGUF	63,387,346,208	4.34	−3%	65.8 GB	No
DeepSeek V3 / R1 (671B)	unsloth/DeepSeek-V3-GGUF9 parts	404,430,187,072	4.73	−2%	415 GB	No

Bits per weight = bytes × 8 ÷ the parameters in the model's safetensors. Total = the file + the FP16 KV cache at 8,192 tokens + 0.5 GB + 10%, in GiB. Each model page lists every quant of its model the same way.

How to reproduce the estimates

Q4_K_M uses an average 4.84 bits per parameter, including GGUF quantization scales. FP8 uses 8 bits and BF16 uses 16 bits. Weight memory is total parameters × bits per parameter ÷ 8; for mixture-of-experts models, all experts count because they must be loaded even though only some run for each token.

The FP16 KV cache is calculated from each model's attention layout: full attention grows with context; sliding-window layers keep only their window; MLA stores a compressed latent; linear or state-space layers do not add a context-growing KV cache. The total is weights + cache + 0.5 GB + 10% of weights and cache. See the KV-cache explanation and calculation source and tests.

The CSV records the weight, cache and overhead components, the three total precisions, model ID, source-check date, architecture summary and per-model URL. Model files were checked on the dates in the CSV; this report was updated 2026-09-29.

Limits

These are estimates, not measured peak GPU usage. vLLM may reserve more memory and llama.cpp can allocate the full configured cache at startup. CUDA graphs, activation buffers, vision encoders and host-side offload can change the real requirement. A “Yes” near a GPU's limit is not a deployment guarantee. Models with a known unmodeled cache optimization carry a note in the table and on their own page. How the estimates compare with public measurements: predicted vs measured.

Reuse the CSV with attribution to this report or the MIT-licensed source. There is no requirement to link back.