Best local LLMs for 32 GB of VRAM

On one 32 GB GPU at Q4_K_M with 32K context, the largest dense model that fits is Gemma 4 31B (31.3B, 23.9 GB, about 40–57 tokens/s on one RTX 5090), the largest MoE is K2-Horizon MoVA 36B-A4B (37.4B total, 4.0B active, 30.3 GB, about 56–96 tokens/s), among models that take 16 GB or more, the longest context goes to a tie between Ornith 1.5 35B-A3B and Qwen3.6 35B-A3B (the full 256K in 28.3 GB), and the largest at Q8_0 is Gemma 4 26B-A4B (MoE, 25.8B total, 4.0B active, 29.6 GB).

Updated , the latest date the data of a model listed here was checked; 69 models in total. Ordered by memory and size, not quality. Total = weights + FP16 KV cache for one request + 0.5 GB + 10% overhead; GB = GiB. A model counts as fitting when it leaves at least 0.5 GB free, as on the Can I run it? pages. Speeds are bandwidth estimates for one request with the 32K context full, on one RTX 5090 (1792 GB/s), not measurements. Not sure how much VRAM you have? Detect your GPU.

gpt-oss-20b is listed at MXFP4, the format it is published in, in the Q4_K_M lists and not in the Q8_0 list: its GGUF quants keep the experts in MXFP4, so every type is about the same size.

Top picks for 32 GB

Pick Model Parameters Active VRAM Tokens/s (RTX 5090) GPQA (reported)
Largest dense (Q4_K_M) Gemma 4 31B 31.3B dense 23.9 GB 40–57 84.3
Largest MoE (Q4_K_M) K2-Horizon MoVA 36B-A4B (MoE) 37.4B 4.0B 30.3 GB 56–96 80.8
Longest context, 16 GB+ models (Q4_K_M) Ornith 1.5 35B-A3B (MoE) 36.0B 3.0B 256K (full), 28.3 GB, tied with Qwen3.6 35B-A3B 163–305 89.2
Largest at Q8_0 Gemma 4 26B-A4B (MoE) 25.8B 4.0B 29.6 GB 89–158 82.3

GPQA is graduate-level science multiple choice (four options, about 25% by guessing). Each score is vendor-reported in the maker’s own model card or report; click a number for the source (read 2026-09-30). We have not rerun them, and reasoning effort and tool use differ between makers, so a gap of a few points means little. — means none was published. 31 models have one.

What is the best local LLM for 32 GB VRAM?

By size, the largest model that fits one 32 GB GPU at Q4_K_M with 32K context is K2-Horizon MoVA 36B-A4B (MoE, 37.4B total, 4.0B active, 30.3 GB, 1.7 GB spare); 38 of the 69 models we track fit. This page ranks by memory and size, not by benchmark scores; the GPQA column is each maker's own reported score, for reference only.

Can 32 GB run a 70B model?

Not on one card: Llama 3.1 70B (70.6B) needs 55.2 GB at Q4_K_M with 32K context, or 2 × 32 GB cards, and still 41.3 GB at IQ3_XXS.

Every step is on the Llama 3.1 70B VRAM page.

Is Q8_0 or a bigger model better on 32 GB?

On one 32 GB GPU with 32K context, the largest model at Q8_0 is Gemma 4 26B-A4B (MoE, 25.8B total, 4.0B active, 29.6 GB, about 89–158 tokens/s) and at Q4_K_M it is K2-Horizon MoVA 36B-A4B (MoE, 37.4B total, 4.0B active, 30.3 GB, about 56–96 tokens/s), 1.5× the total parameters. Q8_0 stores about 8.5 bits per weight and Q4_K_M 4.84; our data covers memory and speed, not output quality, so this page does not score the trade-off.

Bits per weight for every GGUF type are in GGUF quantization explained.

Newest models that fit 32 GB

Sorted by the date each model's data was added or checked, newest first (Q4_K_M, 32K context). New models appear here as they are added.

Model Added / checked Parameters Active VRAM (GB) Tokens/s GPQA (reported)
LightOnOCR-3 4B 4.5B dense 4.41 186–299 —
ThinkingCap Qwen3.8 27B 27.8B dense 19.92 48–68 —
LensVLM 9B 9.4B dense 7.43 120–181 —
Qwen3-Coder 30B-A3B (MoE) 30.5B 3.3B 22.72 89–158 —
Ornith 1.0 35B (MoE) 35.1B 3.0B 22.95 163–305 —
Ornith 1.0 9B 9.4B dense 7.43 120–181 —
Granite 4.2 30B 29.3B dense 27.45 35–50 66.41
Granite 4.2 8B 8.8B dense 11.45 81–118 64.14

All 38 models that fit 32 GB at Q4_K_M

Model Parameters Active VRAM (GB) Spare (GB) Tokens/s GPQA (reported) Calculator
K2-Horizon MoVA 36B-A4B (MoE) 37.4B 4.0B 30.31 1.69 56–96 80.8 Open
Ornith 1.5 35B-A3B (MoE) 36.0B 3.0B 23.47 8.53 163–305 89.2 Open
Qwen3.6 35B-A3B (MoE) 36.0B 3.0B 23.47 8.53 163–305 86.0 Open
Ornith 1.0 35B (MoE) 35.1B 3.0B 22.95 9.05 163–305 — Open
LLM-jp-4.1 32B-A3B Thinking (MoE) 32.1B 3.8B 22.62 9.38 102–182 — Open
Nemotron 3 Nano 30B-A3B (MoE) 31.6B 3.5B 20.28 11.72 172–324 73.0 Open
Gemma 4 31B 31.3B dense 23.92 8.08 40–57 84.3 Open
GLM-4.7 Flash 31.2B 3.0B 21.67 10.33 122–222 75.2 Open
Xing 4.0 29B-A4B (MoE) 31.2B 4.0B 21.39 10.61 114–205 — Open
Qwen3 30B-A3B (MoE) 30.5B 3.3B 22.72 9.28 89–158 — Open
Qwen3-Coder 30B-A3B (MoE) 30.5B 3.3B 22.72 9.28 89–158 — Open
Muse Glimmer 30B 29.8B dense 19.51 12.49 49–70 — Open
Granite 4.2 30B 29.3B dense 27.45 4.55 35–50 66.41 Open
Qwen3.6 27B 27.8B dense 19.92 12.08 48–68 87.8 Open
Qwen3.8 27B 27.8B dense 19.92 12.08 48–68 89.2 Open
ThinkingCap Qwen3.8 27B 27.8B dense 19.92 12.08 48–68 — Open
Hemmingway-1 27B 27.3B dense 19.63 12.37 49–69 — Open
Gemma 4 26B-A4B (MoE) 25.8B 4.0B 17.50 14.50 128–233 82.3 Open
gpt-oss-20b MXFP4 20.9B 3.6B 15.44 16.56 134–246 71.5 Open
Gemma 4 12B 12.0B dense 8.98 23.02 101–150 78.8 Open
ZDTaichu 5.0 9B 9.8B dense 7.67 24.33 116–175 — Open
Ornith 1.5 9B 9.7B dense 7.58 24.42 117–177 86.4 Open
Qwen3.5 9B 9.7B dense 7.58 24.42 117–177 81.7 Open
LensVLM 9B 9.4B dense 7.43 24.57 120–181 — Open
MiMo V2.6 Distill Qwen 9B 9.4B dense 7.43 24.57 120–181 — Open
Ornith 1.0 9B 9.4B dense 7.43 24.57 120–181 — Open
Granite 4.2 8B 8.8B dense 11.45 20.55 81–118 64.14 Open
LFM2.5 8B-A1B (MoE) 8.5B 1.5B 6.16 25.84 254–510 — Open
Qwen3 8B 8.2B dense 10.53 21.47 87–128 — Open
Llama 3.1 8B 8.0B dense 9.88 22.12 93–137 30.4 Open
Gemma 4 E4B 8.0B dense 6.05 25.95 143–221 58.6 Open
Ling 3.0 Tiny 7.9B-A1.3B (MoE) 7.9B 1.3B 5.62 26.38 295–613 — Open
LightOnOCR-3 4B 4.5B dense 4.41 27.59 186–299 — Open
Spark-X2.5 4B 4.1B dense 4.40 27.60 186–300 67.4 Open
Nemotron 3 Nano 4B 4.0B dense 3.51 28.49 223–372 53.2 Open
Granite 4.2 3B 3.7B dense 5.52 26.48 155–241 54.8 Open
MiniCPM5 2B 2.5B dense 3.50 28.50 223–373 — Open
Limite 1B Violetto 1.0B dense 1.62 30.38 383–761 — Open

All 20 models that fit 32 GB at Q8_0

Model Parameters Active VRAM (GB) Spare (GB) Tokens/s GPQA (reported) Calculator
Gemma 4 26B-A4B (MoE) 25.8B 4.0B 29.60 2.40 89–158 82.3 Open
Gemma 4 12B 12.0B dense 14.58 17.42 65–93 78.8 Open
ZDTaichu 5.0 9B 9.8B dense 12.26 19.74 76–111 — Open
Ornith 1.5 9B 9.7B dense 12.11 19.89 77–112 86.4 Open
Qwen3.5 9B 9.7B dense 12.11 19.89 77–112 81.7 Open
LensVLM 9B 9.4B dense 11.84 20.16 79–114 — Open
MiMo V2.6 Distill Qwen 9B 9.4B dense 11.84 20.16 79–114 — Open
Ornith 1.0 9B 9.4B dense 11.84 20.16 79–114 — Open
Granite 4.2 8B 8.8B dense 15.57 16.43 61–87 64.14 Open
LFM2.5 8B-A1B (MoE) 8.5B 1.5B 10.13 21.87 192–367 — Open
Qwen3 8B 8.2B dense 14.37 17.63 66–95 — Open
Llama 3.1 8B 8.0B dense 13.64 18.36 69–100 30.4 Open
Gemma 4 E4B 8.0B dense 9.80 22.20 93–138 58.6 Open
Ling 3.0 Tiny 7.9B-A1.3B (MoE) 7.9B 1.3B 9.32 22.68 223–436 — Open
LightOnOCR-3 4B 4.5B dense 6.54 25.46 134–205 — Open
Spark-X2.5 4B 4.1B dense 6.33 25.67 137–211 67.4 Open
Nemotron 3 Nano 4B 4.0B dense 5.38 26.62 158–247 53.2 Open
Granite 4.2 3B 3.7B dense 7.23 24.77 122–186 54.8 Open
MiniCPM5 2B 2.5B dense 4.68 27.32 177–283 — Open
Limite 1B Violetto 1.0B dense 2.11 29.89 323–600 — Open

Macs and CPU-only PCs with about 32 GB to use

These machines leave a model between 32 GB and the next size up: on a Mac, the unified memory macOS lets the GPU use by default (about two thirds up to 32 GB, 75% above; sudo sysctl iogpu.wired_limit_mb raises it), on a CPU-only PC the RAM minus about 6 GB for the system. The lists above apply to them too; speed follows each one's bandwidth, and a CPU is also much slower at reading long prompts.

Machine Usable memory Bandwidth Models at Q4_K_M (8K) Biggest at 32K Tokens/s
M4 Pro Mac (48 GB) 36 GB 273 GB/s 39 K2-Horizon MoVA 36B-A4B 9.1–15
M5 Pro Mac (48 GB) 36 GB 307 GB/s 39 K2-Horizon MoVA 36B-A4B 10–17
M4 Max Mac (48 GB) 36 GB 546 GB/s 39 K2-Horizon MoVA 36B-A4B 18–30
M5 Max Mac (48 GB) 36 GB 614 GB/s 39 K2-Horizon MoVA 36B-A4B 20–34

Other VRAM sizes

All numbers, including 8K and 128K context, are in the open dataset, with summary figures on LLM VRAM statistics; any other setting can be worked out in the LLM VRAM Calculator.