What LLMs can an RTX 5090 run?

An RTX 5090 (32 GB) runs 39 of the 69 open models tracked here at Q4_K_M with 8K context; the largest is K2-Horizon MoVA 36B-A4B (37.4B parameters, 25.4 GB). It reads memory at 1,792 GB/s. Each fit leaves at least 0.5 GB free on the RTX 5090 (why a card does not give a model all its memory).

Not sure your PC has an RTX 5090? Detect your GPU in the browser.

Can I run it on an RTX 5090 with 32K context?

Gemma 4 26B-A4B: Yes, up to Q8_0
Gemma 4 31B: Yes, up to Q6_K
Qwen3.8 27B: Yes, up to FP8
gpt-oss-20b: Yes, MXFP4
Llama 3.1 8B: Yes, up to BF16
gpt-oss-120b: --n-cpu-moe 19
Qwen3.6 35B-A3B: Yes, up to Q6_K
Qwen3.6 27B: Yes, up to FP8
GLM-4.7 Flash: Yes, up to Q6_K
Qwen3.8 Flash Next: --n-cpu-moe 31
Nemotron 3 Nano 30B-A3B: Yes, up to Q6_K
Qwen3-Coder-Next: --n-cpu-moe 17
Llama 3.1 70B: No, 2 cards

Models that fit on one RTX 5090

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first. gpt-oss-20b (14.8 GB, 158–295 tokens/s on the RTX 5090) is listed at MXFP4 only, as published.

Model	Best precision	Memory	Longest context	Tokens/s
K2-Horizon MoVA 36B-A4B 37.4B, 4B active	GGUF Q5_K_M	29.3 GB	18K	102–183
Ornith 1.5 35B-A3B 36.0B, 3B active	GGUF Q6_K	30.9 GB	37K	157–291
Qwen3.6 35B-A3B 36.0B, 3B active	GGUF Q6_K	30.9 GB	37K	157–291
Ornith 1.0 35B 35.1B, 3B active	GGUF Q6_K	30.2 GB	69K	157–291
LLM-jp-4.1 32B-A3B Thinking 32.1B, 3.8B active	GGUF Q6_K	28.0 GB	58K	120–217
Nemotron 3 Nano 30B-A3B 31.6B, 3.5B active	GGUF Q6_K	27.1 GB	256K (full)	144–266
Gemma 4 31B 31.3B	GGUF Q6_K	28.7 GB	39K	34–48
GLM-4.7 Flash 31.2B, 3B active	GGUF Q6_K	27.2 GB	83K	145–267
Xing 4.0 29B-A4B 31.2B, 4B active	GGUF Q6_K	27.1 GB	98K	120–218
Qwen3-Coder 30B-A3B 30.5B, 3.3B active	GGUF Q6_K	27.0 GB	51K	125–226
Qwen3 30B-A3B 30.5B, 3.3B active	GGUF Q6_K	27.0 GB	40K (full)	125–226
Muse Glimmer 30B 29.8B	FP8 / INT8	31.2 GB	27K	31–44
Granite 4.2 30B 29.3B	GGUF Q6_K	27.3 GB	23K	36–50
ThinkingCap Qwen3.8 27B 27.8B	GGUF Q8_0	31.3 GB	10K	31–44
Qwen3.6 27B 27.8B	GGUF Q8_0	31.3 GB	10K	31–44
Qwen3.8 27B 27.8B	GGUF Q8_0	31.3 GB	10K	31–44
Hemmingway-1 27B 27.3B	GGUF Q8_0	30.8 GB	18K	32–44
Gemma 4 26B-A4B 25.8B, 4B active	GGUF Q8_0	29.1 GB	120K	97–173
gpt-oss-20b 20.9B, 3.6B active	As published (MXFP4)	14.8 GB	128K (full)	158–295
Gemma 4 12B 12.0B	FP16 / BF16	25.7 GB	256K (full)	38–53
ZDTaichu 5.0 9B 9.8B	FP16 / BF16	20.8 GB	128K (full)	46–65
Ornith 1.5 9B 9.7B	FP16 / BF16	20.6 GB	256K (full)	47–66
Qwen3.5 9B 9.7B	FP16 / BF16	20.6 GB	256K (full)	47–66
LensVLM 9B 9.4B	FP16 / BF16	20.1 GB	256K (full)	48–68
Ornith 1.0 9B 9.4B	FP16 / BF16	20.1 GB	256K (full)	48–68
MiMo V2.6 Distill Qwen 9B 9.4B	FP16 / BF16	20.1 GB	256K (full)	48–68
Bespoke Nimble 9B 9.0B	FP16 / BF16	19.1 GB	8K (full)	50–71
Granite 4.2 8B 8.8B	FP16 / BF16	19.9 GB	75K	48–69
LFM2.5 8B-A1B 8.5B, 1.5B active	FP16 / BF16	18.0 GB	128,000 (full)	138–252
Qwen3 8B 8.2B	FP16 / BF16	18.5 GB	40K (full)	52–74
Llama 3.1 8B 8.0B	FP16 / BF16	18.1 GB	105K	53–75
Gemma 4 E4B 8.0B	FP16 / BF16	17.1 GB	128K (full)	56–80
Ling 3.0 Tiny 7.9B-A1.3B 7.9B, 1.3B active	FP16 / BF16	16.7 GB	128K (full)	155–289
LightOnOCR-3 4B 4.5B	FP16 / BF16	10.1 GB	256K (full)	91–134
Spark-X2.5 4B 4.1B	FP16 / BF16	9.35 GB	580K	97–144
Nemotron 3 Nano 4B 4.0B	FP16 / BF16	8.78 GB	256K (full)	103–154
Granite 4.2 3B 3.7B	FP16 / BF16	8.69 GB	128K (full)	104–155
MiniCPM5 2B 2.5B	FP16 / BF16	6.02 GB	128K (full)	144–222
Limite 1B Violetto 1.0B	FP16 / BF16	2.79 GB	128K (full)	265–462

Too big for one RTX 5090 at 8K context

How many RTX 5090 cards these need at Q4_K_M (gpt-oss-120b at MXFP4) with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.

Llama 3.1 70B: 2 cards
Kolibri-1: 2 cards
Qwen3-Coder-Next: 2 cards
AliceAI Foundation 80B-A3B: 2 cards
gpt-oss-120b: 4 cards
Nemotron 3 Super 120B-A12B: 4 cards
Nemotron 3.5 Super VL 120B-A12B: 4 cards
Qwen3.5 122B-A10B: 4 cards
Mistral Medium 3.5 128B: 4 cards
Qwen3.8 Flash Next: 4 cards
Step 3.7 Flash 196B-A11B: 4 cards
MiniMax M2.7: 8 cards
DeepSeek V4 Flash: 8 cards
DeepSeek V4 Flash 0731: 8 cards
MiMo V2.6 Flash: 8 cards
IQuest-Q1 320B-A15B: 8 cards
GLM-5.3 Flash: 8 cards
MiniMax M3: more than 8 cards
Ling 3.1 Flash 560B-A25B: more than 8 cards
DeepSeek V3 / R1: more than 8 cards
DeepSeek V3.2: more than 8 cards
GLM-5.3: more than 8 cards
GLM-5.2: more than 8 cards
DeepSeek V4.1 Flash 1017: more than 8 cards
DeepSeek V4.1 Flash: more than 8 cards
Hy4 Preview 770B-A49B: more than 8 cards
MiMo V2.6 Pro: more than 8 cards
DeepSeek V4 Pro: more than 8 cards
Qwen3.8 2.4T-A95B: more than 8 cards
Kimi K3: more than 8 cards

Best models for RTX 5090 with room for long context

The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the card takes at that precision and the writing speed with that context in the cache, from 1,792 GB/s of bandwidth.

Model	Context	Memory	Left over	Longest context	Tokens/s
K2-Horizon MoVA 36B-A4B 37.4B	32K	30.3 GB	1.69 GB	37K	56–96
Ornith 1.5 35B-A3B 36.0B	32K	23.5 GB	8.53 GB	256K (full)	163–305
Qwen3.6 35B-A3B 36.0B	32K	23.5 GB	8.53 GB	256K (full)	163–305
Ornith 1.0 35B 35.1B	32K	22.9 GB	9.05 GB	256K (full)	163–305
LLM-jp-4.1 32B-A3B Thinking 32.1B	32K	22.6 GB	9.38 GB	64K (full)	102–182

llama-server commands for one RTX 5090

GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on the RTX 5090. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.

K2-Horizon MoVA 36B-A4B, Q4_K_M with 37K tokens: 50–87 tokens/s on the RTX 5090

llama-server -hf IFM/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M -c 37888

K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf, 22.4 GB: 31.3 GB used and 677 MB free of the RTX 5090's 32 GB, 50–87 tokens/s once the 37K cache is full.

Ornith 1.5 35B-A3B, Q4_K_M with 256K tokens: 67–117 tokens/s on the RTX 5090

llama-server -hf bartowski/Ornith-1.5-35B-A3B-GGUF:Q4_K_M -c 262144

Ornith-1.5-35B-A3B-Q4_K_M.gguf, 21.9 GB: 28.4 GB used and 3.60 GB free of the RTX 5090's 32 GB, 67–117 tokens/s once the 256K cache is full.

How fast the RTX 5090 writes as the context fills

Tokens per second at Q4_K_M for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the card holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 1,792 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside.

Model	8K	32K	Longest	At the longest	Q8_0 at 8K	Bandwidth ceiling at 8K	1,000-token reply at 32K
K2-Horizon MoVA 36B-A4B	111–200	56–96	37K	50–87	Does not fit	445	10–18 s
Ornith 1.5 35B-A3B	193–369	163–305	256K	67–117	Does not fit	904	3–6 s
Qwen3.6 35B-A3B	193–369	163–305	256K	67–117	Does not fit	904	3–6 s
Ornith 1.0 35B	193–369	163–305	256K	67–117	Does not fit	904	3–6 s
LLM-jp-4.1 32B-A3B Thinking	147–271	102–182	64K	72–127	Does not fit	628	5–10 s
Nemotron 3 Nano 30B-A3B	181–343	172–324	256K	119–215	Does not fit	827	3–6 s
Gemma 4 31B	44–62	40–57	120K	31–43	Does not fit	86	18–25 s
GLM-4.7 Flash	175–331	122–222	198K	40–68	Does not fit	793	5–8 s
Xing 4.0 29B-A4B	149–276	114–205	241K	37–63	Does not fit	641	5–9 s
Qwen3-Coder 30B-A3B	149–276	89–158	117K	37–63	Does not fit	640	6–11 s

What more cards or MoE offload unlock

One, two and four RTX 5090 cards in one tensor-parallel group (vLLM or SGLang), with the models each step adds at Q4_K_M and 8K context and their tokens per second on that group, and the median on-demand rent for that many cards (getdeploying.com, 2026-09-29).

Setup	Memory	Bandwidth	Models at Q4_K_M	Rent per hour	Adds (tokens/s)
RTX 5090	32 GB	1,792 GB/s	39	$0.69	This page
2× RTX 5090	64 GB	3,584 GB/s	43	$1.38	Llama 3.1 70B 38–55, Kolibri-1 175–382, Qwen3-Coder-Next 186–412, AliceAI Foundation 80B-A3B 186–412
4× RTX 5090	128 GB	7,168 GB/s	50	$2.76	gpt-oss-120b 201–457, Nemotron 3 Super 120B-A12B 145–299, Nemotron 3.5 Super VL 120B-A12B 145–299, Qwen3.5 122B-A10B 156–328, Mistral Medium 3.5 128B 42–62, Qwen3.8 Flash Next 189–422, Step 3.7 Flash 196B-A11B 146–302

Every mixture-of-experts GGUF file the --n-cpu-moe planner has measured, on one RTX 5090 with 32K tokens of context (or the model's whole window): the smallest --n-cpu-moe that fits the card (0 means the whole file fits), the memory on the card and in system RAM, and tokens per second from 1,792 GB/s of card bandwidth with DDR5-5600, 2 channels (89.6 GB/s) or DDR4-3200, 2 channels (51.2 GB/s). 4 of these models do not fit whole and need offload.

Model	GGUF	--n-cpu-moe	On the card	In system RAM	Tokens/s, DDR5	Tokens/s, DDR4
gpt-oss-20b	MXFP4	0	12.5 GB	587 MB	128–233	128–233
Gemma 4 26B-A4B	UD-Q4_K_M	0	17.0 GB	748 MB	120–217	120–217
Qwen3 30B-A3B	Q4_K_M	0	21.1 GB	167 MB	90–160	90–160
Qwen3 30B-A3B	Q8_0	4	31.5 GB	2.70 GB	52–90	42–72
GLM-4.7 Flash	UD-Q4_K_XL	0	18.8 GB	170 MB	107–193	107–193
GLM-4.7 Flash	Q4_K_M	0	19.5 GB	170 MB	108–194	108–194
Nemotron 3 Nano 30B-A3B	Q4_K_M	0	23.8 GB	231 MB	150–279	150–279
Qwen3.6 35B-A3B	UD-IQ4_XS	0	17.6 GB	515 MB	136–248	136–248
Ornith 1.5 35B-A3B	Q4_K_M	0	21.6 GB	273 MB	154–286	154–286
Qwen3.6 35B-A3B	UD-Q4_K_M	0	21.7 GB	515 MB	131–239	131–239
Qwen3.6 35B-A3B	Q8_0	5	31.5 GB	4.49 GB	75–131	58–101
Ornith 1.5 35B-A3B	Q8_0	6	31.5 GB	5.28 GB	69–121	53–91
Qwen3-Coder-Next	IQ4_XS	12	31.8 GB	9.73 GB	70–123	50–87
Qwen3-Coder-Next	Q4_K_M	17	31.3 GB	15.7 GB	52–91	36–61
Qwen3-Coder-Next	UD-Q4_K_XL	18	31.3 GB	16.6 GB	47–81	33–55
gpt-oss-120b	MXFP4	19	30.6 GB	30.6 GB	22–37	13–23
Qwen3.5 122B-A10B	Q4_K_M	30	31.5 GB	41.5 GB	15–26	10–16
Qwen3.8 Flash Next	UD-Q2_K_XL	18	31.8 GB	16.6 GB	43–74	31–52
Qwen3.8 Flash Next	UD-IQ3_XXS	20	31.8 GB	19.5 GB	38–65	27–45
Qwen3.8 Flash Next	GSQ-RCO IQ3_S	23	31.3 GB	21.5 GB	35–60	25–42
Qwen3.8 Flash Next	UD-Q3_K_XL	25	31.1 GB	27.6 GB	30–50	20–34
Qwen3.8 Flash Next	UD-IQ4_XS	26	31.7 GB	30.5 GB	28–47	19–32
Qwen3.8 Flash Next	UD-Q4_K_XL	31	31.6 GB	47.0 GB	20–34	13–22
Qwen3.8 Flash Next	UD-Q5_K_XL	35	31.0 GB	67.5 GB	15–26	10–16

In system RAM counts the offloaded experts plus the input embedding (token_embd), which llama.cpp keeps in RAM in any case.

Run locally or rent an RTX 5090?

Renting an RTX 5090 costs a median of $0.69 an hour on demand, across the providers getdeploying.com tracks (checked 2026-09-29), so every $1,000 of purchase price equals about 1,449 hours of rent, 60 days around the clock. At that rate K2-Horizon MoVA 36B-A4B at Q4_K_M with 32K tokens, the biggest model here that fits with a long context, writes 56–96 tokens/s for one request: about $2.0–3.4 per million generated tokens. Electricity and resale value are left out, and no card prices are listed because 2026 street prices vary widely between sellers; compare current ones before you decide.

Serving with vLLM on the RTX 5090

The KV cache pool vLLM builds at its default --gpu-memory-utilization 0.92 and --max-model-len 32768, weights as published, from the vLLM KV cache calculator. The pool is shared by every request; the last column assumes 4,096 tokens each.

Model	KV cache per GPU	Pool (tokens)	At 32K	Requests at 4K
gpt-oss-20b	13.8 GB	533,595	16.28x	73
Qwen3 8B	12.1 GB	88,272	2.69x	21

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Nearby GPUs and setups

How many of the tracked models each one holds at Q4_K_M with 8K context, against 39 here.

Setup	How it relates	Memory	Bandwidth	Models at Q4_K_M
RTX 4090	Next size down	24 GB	1,008 GB/s	38 (−1)
M4 Pro Mac (64 GB)	Next size up	48 GB usable	273 GB/s	40 (+1)
2× RTX 5090	More of this card	64 GB	3,584 GB/s	43 (+4)

Model numbers read from Hugging Face on 2026-10-20.