What LLMs can an RTX 4090 run?

An RTX 4090 (24 GB) runs 38 of the 69 open models tracked here at Q4_K_M with 8K context; the largest is Ornith 1.5 35B-A3B (36.0B parameters, 23.0 GB). It reads memory at 1,008 GB/s. 1 more fits at a lower precision. Each fit leaves at least 0.5 GB free on the RTX 4090 (why a card does not give a model all its memory).

Not sure your PC has an RTX 4090? Detect your GPU in the browser.

Can I run it on an RTX 4090 with 32K context?

Gemma 4 26B-A4B: Yes, up to Q6_K
Gemma 4 31B: Q4_K_M, 26K context
Qwen3.8 27B: Yes, up to Q5_K_M
gpt-oss-20b: Yes, MXFP4
Llama 3.1 8B: Yes, up to BF16
gpt-oss-120b: --n-cpu-moe 24
Qwen3.6 35B-A3B: Yes, Q4_K_M
Qwen3.6 27B: Yes, up to Q5_K_M
Gemma 4 12B: Yes, up to Q8_0
GLM-4.7 Flash: Yes, Q4_K_M
Qwen3.8 Flash Next: --n-cpu-moe 37
Nemotron 3 Nano 30B-A3B: Yes, Q4_K_M
Qwen3-Coder-Next: --n-cpu-moe 26
Llama 3.1 70B: No, 4 cards

Models that fit on one RTX 4090

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first. gpt-oss-20b (14.8 GB, 99–177 tokens/s on the RTX 4090) is listed at MXFP4 only, as published.

Model	Best precision	Memory	Longest context	Tokens/s
K2-Horizon MoVA 36B-A4B 37.4B, 4B active	GGUF Q3_K_M	20.9 GB	20K	75–132
Ornith 1.5 35B-A3B 36.0B, 3B active	GGUF Q4_K_M	23.0 GB	33K	124–226
Qwen3.6 35B-A3B 36.0B, 3B active	GGUF Q4_K_M	23.0 GB	33K	124–226
Ornith 1.0 35B 35.1B, 3B active	GGUF Q4_K_M	22.4 GB	57K	124–226
LLM-jp-4.1 32B-A3B Thinking 32.1B, 3.8B active	GGUF Q4_K_M	21.0 GB	44K	91–162
Nemotron 3 Nano 30B-A3B 31.6B, 3.5B active	GGUF Q5_K_M	23.5 GB	10K	101–181
Gemma 4 31B 31.3B	GGUF Q4_K_M	21.9 GB	26K	26–36
GLM-4.7 Flash 31.2B, 3B active	GGUF Q4_K_M	20.3 GB	64K	111–201
Xing 4.0 29B-A4B 31.2B, 4B active	GGUF Q4_K_M	20.2 GB	75K	93–165
Qwen3-Coder 30B-A3B 30.5B, 3.3B active	GGUF Q5_K_M	23.5 GB	7K	84–148
Qwen3 30B-A3B 30.5B, 3.3B active	GGUF Q5_K_M	23.5 GB	8K	84–148
Muse Glimmer 30B 29.8B	GGUF Q5_K_M	22.3 GB	91K	25–35
Granite 4.2 30B 29.3B	GGUF Q4_K_M	20.8 GB	17K	27–37
ThinkingCap Qwen3.8 27B 27.8B	GGUF Q5_K_M	21.2 GB	41K	26–37
Qwen3.6 27B 27.8B	GGUF Q5_K_M	21.2 GB	41K	26–37
Qwen3.8 27B 27.8B	GGUF Q5_K_M	21.2 GB	41K	26–37
Hemmingway-1 27B 27.3B	GGUF Q5_K_M	20.9 GB	45K	27–37
Gemma 4 26B-A4B 25.8B, 4B active	GGUF Q6_K	22.7 GB	46K	72–126
gpt-oss-20b 20.9B, 3.6B active	As published (MXFP4)	14.8 GB	128K (full)	99–177
Gemma 4 12B 12.0B	GGUF Q8_0	14.2 GB	256K (full)	39–55
ZDTaichu 5.0 9B 9.8B	FP16 / BF16	20.8 GB	85K	27–37
Ornith 1.5 9B 9.7B	FP16 / BF16	20.6 GB	93K	27–38
Qwen3.5 9B 9.7B	FP16 / BF16	20.6 GB	93K	27–38
LensVLM 9B 9.4B	FP16 / BF16	20.1 GB	108K	28–39
Ornith 1.0 9B 9.4B	FP16 / BF16	20.1 GB	108K	28–39
MiMo V2.6 Distill Qwen 9B 9.4B	FP16 / BF16	20.1 GB	108K	28–39
Bespoke Nimble 9B 9.0B	FP16 / BF16	19.1 GB	8K (full)	29–41
Granite 4.2 8B 8.8B	FP16 / BF16	19.9 GB	28K	28–39
LFM2.5 8B-A1B 8.5B, 1.5B active	FP16 / BF16	18.0 GB	128,000 (full)	85–150
Qwen3 8B 8.2B	FP16 / BF16	18.5 GB	40K (full)	30–42
Llama 3.1 8B 8.0B	FP16 / BF16	18.1 GB	47K	31–43
Gemma 4 E4B 8.0B	FP16 / BF16	17.1 GB	128K (full)	33–46
Ling 3.0 Tiny 7.9B-A1.3B 7.9B, 1.3B active	FP16 / BF16	16.7 GB	128K (full)	97–173
LightOnOCR-3 4B 4.5B	FP16 / BF16	10.1 GB	256K (full)	54–78
Spark-X2.5 4B 4.1B	FP16 / BF16	9.35 GB	373K	59–84
Nemotron 3 Nano 4B 4.0B	FP16 / BF16	8.78 GB	256K (full)	62–89
Granite 4.2 3B 3.7B	FP16 / BF16	8.69 GB	128K (full)	63–90
MiniCPM5 2B 2.5B	FP16 / BF16	6.02 GB	128K (full)	89–131
Limite 1B Violetto 1.0B	FP16 / BF16	2.79 GB	128K (full)	181–289

Too big for one RTX 4090 at 8K context

How many RTX 4090 cards these need at Q4_K_M (gpt-oss-120b at MXFP4) with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.

Llama 3.1 70B: 2 cards
Kolibri-1: 4 cards
Qwen3-Coder-Next: 4 cards
AliceAI Foundation 80B-A3B: 4 cards
gpt-oss-120b: 4 cards
Nemotron 3 Super 120B-A12B: 4 cards
Nemotron 3.5 Super VL 120B-A12B: 4 cards
Qwen3.5 122B-A10B: 4 cards
Mistral Medium 3.5 128B: 4 cards
Qwen3.8 Flash Next: 8 cards
Step 3.7 Flash 196B-A11B: 8 cards
MiniMax M2.7: 8 cards
DeepSeek V4 Flash: 8 cards
DeepSeek V4 Flash 0731: more than 8 cards
MiMo V2.6 Flash: more than 8 cards
IQuest-Q1 320B-A15B: more than 8 cards
GLM-5.3 Flash: more than 8 cards
MiniMax M3: more than 8 cards
Ling 3.1 Flash 560B-A25B: more than 8 cards
DeepSeek V3 / R1: more than 8 cards
DeepSeek V3.2: more than 8 cards
GLM-5.3: more than 8 cards
GLM-5.2: more than 8 cards
DeepSeek V4.1 Flash 1017: more than 8 cards
DeepSeek V4.1 Flash: more than 8 cards
Hy4 Preview 770B-A49B: more than 8 cards
MiMo V2.6 Pro: more than 8 cards
DeepSeek V4 Pro: more than 8 cards
Qwen3.8 2.4T-A95B: more than 8 cards
Kimi K3: more than 8 cards

Best models for RTX 4090 with room for long context

The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the card takes at that precision and the writing speed with that context in the cache, from 1,008 GB/s of bandwidth.

Model	Context	Memory	Left over	Longest context	Tokens/s
Ornith 1.5 35B-A3B 36.0B	32K	23.5 GB	542 MB	33K	103–184
Qwen3.6 35B-A3B 36.0B	32K	23.5 GB	542 MB	33K	103–184
Ornith 1.0 35B 35.1B	32K	22.9 GB	1.05 GB	57K	103–184
LLM-jp-4.1 32B-A3B Thinking 32.1B	32K	22.6 GB	1.38 GB	44K	62–107
Nemotron 3 Nano 30B-A3B 31.6B	32K	20.3 GB	3.72 GB	256K (full)	109–196

llama-server commands for one RTX 4090

GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on the RTX 4090. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.

Ornith 1.5 35B-A3B, Q4_K_M with 27K tokens: 107–191 tokens/s on the RTX 4090

llama-server -hf bartowski/Ornith-1.5-35B-A3B-GGUF:Q4_K_M -c 27648

Ornith-1.5-35B-A3B-Q4_K_M.gguf, 21.9 GB: 23.5 GB used and 534 MB free of the RTX 4090's 24 GB, 107–191 tokens/s once the 27K cache is full.

Qwen3.6 35B-A3B, Q4_K_M with 33K tokens: 102–183 tokens/s on the RTX 4090

llama-server -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M -c 33792

Qwen3.6-35B-A3B-Q4_K_M.gguf, 20.4 GB: 23.5 GB used and 520 MB free of the RTX 4090's 24 GB, 102–183 tokens/s once the 33K cache is full.

Ornith 1.0 35B, Q4_K_M with 50K tokens: 91–162 tokens/s on the RTX 4090

llama-server -hf bartowski/deepreinforce-ai_Ornith-1.0-35B-GGUF:Q4_K_M -c 51200

deepreinforce-ai_Ornith-1.0-35B-Q4_K_M.gguf, 21.4 GB: 23.5 GB used and 523 MB free of the RTX 4090's 24 GB, 91–162 tokens/s once the 50K cache is full.

How fast the RTX 4090 writes as the context fills

Tokens per second at Q4_K_M for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the card holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 1,008 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside. With the same memory, RTX 3090, 936 GB/s, writes 6% slower on average; RX 7900 XTX, 960 GB/s, writes 4% slower on average; 2× RTX 3060 12GB, 720 GB/s, writes 29% slower by bandwidth.

Model	8K	32K	Longest	At the longest	Q8_0 at 8K	Bandwidth ceiling at 8K	1,000-token reply at 32K
Ornith 1.5 35B-A3B	124–226	103–184	33K	102–183	Does not fit	508	5–10 s
Qwen3.6 35B-A3B	124–226	103–184	33K	102–183	Does not fit	508	5–10 s
Ornith 1.0 35B	124–226	103–184	57K	87–154	Does not fit	508	5–10 s
LLM-jp-4.1 32B-A3B Thinking	91–162	62–107	44K	53–91	Does not fit	353	9–16 s
Nemotron 3 Nano 30B-A3B	115–208	109–196	256K	72–127	Does not fit	465	5–9 s
GLM-4.7 Flash	111–201	75–131	64K	52–90	Does not fit	446	8–13 s
Xing 4.0 29B-A4B	93–165	69–121	75K	47–81	Does not fit	360	8–14 s
Qwen3-Coder 30B-A3B	93–165	53–92	39K	47–82	Does not fit	360	11–19 s
Qwen3 30B-A3B	93–165	53–92	39K	47–82	Does not fit	360	11–19 s
Muse Glimmer 30B	29–41	29–40	128K	27–37	Does not fit	55	25–35 s

RTX 4090 against the other 24 GB cards

The same models fit on every card with 24 GB, so speed is what separates them. By bandwidth the RTX 4090 is the fastest of the 3 cards with 24 GB at 1,008 GB/s. Over the 12 biggest models that fit at Q4_K_M with 8K context, the RTX 3090 (936 GB/s) writes 6% slower and the RX 7900 XTX (960 GB/s) writes 4% slower than the RTX 4090. Each cell below is the other card's tokens per second minus this one's, midpoints of the estimated ranges.

Model	RTX 4090 tokens/s	RTX 3090	RX 7900 XTX
Ornith 1.5 35B-A3B	124–226	−11	−7.2
Qwen3.6 35B-A3B	124–226	−11	−7.2
Ornith 1.0 35B	124–226	−11	−7.2
LLM-jp-4.1 32B-A3B Thinking	91–162	−8.2	−5.5
Nemotron 3 Nano 30B-A3B	115–208	−10	−6.8
Gemma 4 31B	26–36	−2.1	−1.4
GLM-4.7 Flash	111–201	−9.9	−6.5
Xing 4.0 29B-A4B	93–165	−8.3	−5.5
Qwen3-Coder 30B-A3B	93–165	−8.3	−5.5
Qwen3 30B-A3B	93–165	−8.3	−5.5
Muse Glimmer 30B	29–41	−2.4	−1.6
Granite 4.2 30B	27–37	−2.2	−1.5

Near misses on RTX 4090

Models that miss at Q4_K_M with 32,768 tokens of context (or their whole window), or leave under 0.5 GB free there, but fit with a shorter context or 3-bit weights, or are short by at most a quarter of the memory and fit with MoE offload. Smallest shortfall first.

Model	Short by at Q4_K_M	Fits instead
Gemma 4 31B 31.3B	Tight, 80 MB free	Q4_K_M with 26K context
Granite 4.2 30B 29.3B	3.45 GB	Q4_K_M with 17K context
K2-Horizon MoVA 36B-A4B 37.4B	6.31 GB	GGUF IQ3_XXS with 32K, 22.9 GB

What more cards or MoE offload unlock

One, two and four RTX 4090 cards in one tensor-parallel group (vLLM or SGLang), with the models each step adds at Q4_K_M and 8K context and their tokens per second on that group, and the median on-demand rent for that many cards (getdeploying.com, 2026-09-29).

Setup	Memory	Bandwidth	Models at Q4_K_M	Rent per hour	Adds (tokens/s)
RTX 4090	24 GB	1,008 GB/s	38	$0.53	This page
2× RTX 4090	48 GB	2,016 GB/s	40	$1.06	K2-Horizon MoVA 36B-A4B 98–189, Llama 3.1 70B 23–32
4× RTX 4090	96 GB	4,032 GB/s	48	$2.12	Kolibri-1 183–404, Qwen3-Coder-Next 194–435, AliceAI Foundation 80B-A3B 194–435, gpt-oss-120b 163–348, Nemotron 3 Super 120B-A12B 105–203, Nemotron 3.5 Super VL 120B-A12B 105–203, Qwen3.5 122B-A10B 115–227, Mistral Medium 3.5 128B 25–36

Every mixture-of-experts GGUF file the --n-cpu-moe planner has measured, on one RTX 4090 with 32K tokens of context (or the model's whole window): the smallest --n-cpu-moe that fits the card (0 means the whole file fits), the memory on the card and in system RAM, and tokens per second from 1,008 GB/s of card bandwidth with DDR5-5600, 2 channels (89.6 GB/s) or DDR4-3200, 2 channels (51.2 GB/s). 4 of these models do not fit whole and need offload.

Model	GGUF	--n-cpu-moe	On the card	In system RAM	Tokens/s, DDR5	Tokens/s, DDR4
gpt-oss-20b	MXFP4	0	12.5 GB	587 MB	78–138	78–138
Gemma 4 26B-A4B	UD-Q4_K_M	0	17.0 GB	748 MB	73–128	73–128
Qwen3 30B-A3B	Q4_K_M	0	21.1 GB	167 MB	54–93	54–93
Qwen3 30B-A3B	Q8_0	17	23.8 GB	10.5 GB	22–37	15–26
GLM-4.7 Flash	UD-Q4_K_XL	0	18.8 GB	170 MB	65–113	65–113
GLM-4.7 Flash	Q4_K_M	0	19.5 GB	170 MB	65–114	65–114
Nemotron 3 Nano 30B-A3B	Q4_K_M	0	23.8 GB	231 MB	94–167	94–167
Qwen3.6 35B-A3B	UD-IQ4_XS	0	17.6 GB	515 MB	84–148	84–148
Ornith 1.5 35B-A3B	Q4_K_M	0	21.6 GB	273 MB	96–171	96–171
Qwen3.6 35B-A3B	UD-Q4_K_M	0	21.7 GB	515 MB	81–142	81–142
Qwen3.6 35B-A3B	Q8_0	15	23.5 GB	12.5 GB	36–61	26–43
Ornith 1.5 35B-A3B	Q8_0	16	23.6 GB	13.3 GB	34–59	24–41
Qwen3-Coder-Next	IQ4_XS	22	23.8 GB	17.7 GB	42–71	29–50
Qwen3-Coder-Next	Q4_K_M	26	23.3 GB	23.6 GB	35–59	23–40
Qwen3-Coder-Next	UD-Q4_K_XL	27	23.2 GB	24.8 GB	31–53	21–36
gpt-oss-120b	MXFP4	24	22.7 GB	38.5 GB	17–28	10–17
Qwen3.5 122B-A10B	Q4_K_M	36	23.3 GB	49.7 GB	12–20	8–13
Qwen3.8 Flash Next	UD-Q2_K_XL	27	23.8 GB	24.6 GB	28–47	20–34
Qwen3.8 Flash Next	UD-IQ3_XXS	29	23.3 GB	27.9 GB	25–42	18–30
Qwen3.8 Flash Next	GSQ-RCO IQ3_S	31	23.2 GB	29.6 GB	24–40	17–28
Qwen3.8 Flash Next	UD-Q3_K_XL	32	23.5 GB	35.2 GB	21–35	15–25
Qwen3.8 Flash Next	UD-IQ4_XS	33	23.5 GB	38.6 GB	20–33	14–23
Qwen3.8 Flash Next	UD-Q4_K_XL	37	22.8 GB	55.8 GB	16–26	10–17
Qwen3.8 Flash Next	UD-Q5_K_XL	39	23.4 GB	75.1 GB	13–21	8–14

In system RAM counts the offloaded experts plus the input embedding (token_embd), which llama.cpp keeps in RAM in any case.

Run locally or rent an RTX 4090?

Renting an RTX 4090 costs a median of $0.53 an hour on demand, across the providers getdeploying.com tracks (checked 2026-09-29), so every $1,000 of purchase price equals about 1,887 hours of rent, 79 days around the clock. At that rate Ornith 1.5 35B-A3B at Q4_K_M with 32K tokens, the biggest model here that fits with a long context, writes 103–184 tokens/s for one request: about $0.80–1.4 per million generated tokens. Electricity and resale value are left out, and no card prices are listed because 2026 street prices vary widely between sellers; compare current ones before you decide.

Serving with vLLM on the RTX 4090

The KV cache pool vLLM builds at its default --gpu-memory-utilization 0.92 and --max-model-len 32768, weights as published, from the vLLM KV cache calculator. The pool is shared by every request; the last column assumes 4,096 tokens each.

Model	KV cache per GPU	Pool (tokens)	At 32K	Requests at 4K
gpt-oss-20b	6.6 GB	254,578	7.77x	35
Qwen3 8B	4.9 GB	35,744	1.09x	8

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Nearby GPUs and setups

How many of the tracked models each one holds at Q4_K_M with 8K context, against 38 here.

Setup	How it relates	Memory	Bandwidth	Models at Q4_K_M
RTX 3090	Same memory	24 GB	936 GB/s	38 (same)
RX 7900 XTX	Same memory	24 GB	960 GB/s	38 (same)
2× RTX 3060 12GB	Same memory	24 GB	720 GB/s	38 (same)
RTX 5080 16GB	Next size down	16 GB	960 GB/s	21 (−17)
RTX 5090	Next size up	32 GB	1,792 GB/s	39 (+1)
2× RTX 4090	More of this card	48 GB	2,016 GB/s	40 (+2)

Model numbers read from Hugging Face on 2026-10-20.