What LLMs can an RTX 4070 12GB run?

An RTX 4070 12GB runs 20 of the 69 open models tracked here at Q4_K_M with 8K context; the largest is Gemma 4 12B (12.0B parameters, 8.57 GB). It reads memory at 504 GB/s. Each fit leaves at least 0.5 GB free on the RTX 4070 12GB (why a card does not give a model all its memory).

Not sure your PC has an RTX 4070 12GB? Detect your GPU in the browser.

Can I run it on an RTX 4070 12GB with 32K context?

Gemma 4 26B-A4B: --n-cpu-moe 12
Gemma 4 31B: No, 4 cards
Qwen3.8 27B: No, 2 cards
gpt-oss-20b: --n-cpu-moe 2
Llama 3.1 8B: Yes, up to Q5_K_M
gpt-oss-120b: --n-cpu-moe 31
Qwen3.6 35B-A3B: --n-cpu-moe 22
Qwen3.6 27B: No, 2 cards
Gemma 4 12B: Yes, up to Q5_K_M
GLM-4.7 Flash: --n-cpu-moe 24
Qwen3.8 Flash Next: --n-cpu-moe 45
Nemotron 3 Nano 30B-A3B: --n-cpu-moe 30
Qwen3-Coder-Next: --n-cpu-moe 39
Llama 3.1 70B: No, 8 cards

Models that fit on one RTX 4070 12GB

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first.

Model	Best precision	Memory	Longest context	Tokens/s
Gemma 4 12B 12.0B	GGUF Q6_K	11.2 GB	25K	26–36
ZDTaichu 5.0 9B 9.8B	GGUF Q8_0	11.4 GB	9K	25–35
Ornith 1.5 9B 9.7B	GGUF Q8_0	11.3 GB	14K	25–35
Qwen3.5 9B 9.7B	GGUF Q8_0	11.3 GB	14K	25–35
LensVLM 9B 9.4B	GGUF Q8_0	11.0 GB	21K	26–36
Ornith 1.0 9B 9.4B	GGUF Q8_0	11.0 GB	21K	26–36
MiMo V2.6 Distill Qwen 9B 9.4B	GGUF Q8_0	11.0 GB	21K	26–36
Bespoke Nimble 9B 9.0B	GGUF Q8_0	10.5 GB	8K (full)	27–38
Granite 4.2 8B 8.8B	GGUF Q8_0	11.4 GB	8K	25–35
LFM2.5 8B-A1B 8.5B, 1.5B active	GGUF Q8_0	9.82 GB	128,000 (full)	79–138
Qwen3 8B 8.2B	GGUF Q8_0	10.7 GB	13K	27–37
Llama 3.1 8B 8.0B	GGUF Q8_0	10.3 GB	16K	28–39
Gemma 4 E4B 8.0B	GGUF Q8_0	9.38 GB	128K (full)	31–43
Ling 3.0 Tiny 7.9B-A1.3B 7.9B, 1.3B active	GGUF Q8_0	9.15 GB	128K (full)	91–161
LightOnOCR-3 4B 4.5B	FP16 / BF16	10.1 GB	49K	28–40
Spark-X2.5 4B 4.1B	FP16 / BF16	9.35 GB	63K	31–43
Nemotron 3 Nano 4B 4.0B	FP16 / BF16	8.78 GB	166K	33–46
Granite 4.2 3B 3.7B	FP16 / BF16	8.69 GB	40K	33–46
MiniCPM5 2B 2.5B	FP16 / BF16	6.02 GB	128K (full)	48–68
Limite 1B Violetto 1.0B	FP16 / BF16	2.79 GB	128K (full)	104–156

Too big for one RTX 4070 12GB at 8K context

How many RTX 4070 12GB cards these need at Q4_K_M (gpt-oss-20b and gpt-oss-120b at MXFP4) with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.

gpt-oss-20b: 2 cards
Gemma 4 26B-A4B: 2 cards
Hemmingway-1 27B: 2 cards
ThinkingCap Qwen3.8 27B: 2 cards
Qwen3.6 27B: 2 cards
Qwen3.8 27B: 2 cards
Granite 4.2 30B: 2 cards
Muse Glimmer 30B: 2 cards
Qwen3-Coder 30B-A3B: 2 cards
Qwen3 30B-A3B: 2 cards
Xing 4.0 29B-A4B: 2 cards
GLM-4.7 Flash: 2 cards
Gemma 4 31B: 2 cards
Nemotron 3 Nano 30B-A3B: 2 cards
LLM-jp-4.1 32B-A3B Thinking: 2 cards
Ornith 1.0 35B: 2 cards
Ornith 1.5 35B-A3B: 2 cards
Qwen3.6 35B-A3B: 2 cards
K2-Horizon MoVA 36B-A4B: 4 cards
Llama 3.1 70B: 8 cards
Kolibri-1: 8 cards
Qwen3-Coder-Next: 8 cards
AliceAI Foundation 80B-A3B: 8 cards
gpt-oss-120b: 8 cards
Nemotron 3 Super 120B-A12B: 8 cards
Nemotron 3.5 Super VL 120B-A12B: 8 cards
Qwen3.5 122B-A10B: 8 cards
Mistral Medium 3.5 128B: 8 cards
Qwen3.8 Flash Next: more than 8 cards
Step 3.7 Flash 196B-A11B: more than 8 cards
MiniMax M2.7: more than 8 cards
DeepSeek V4 Flash: more than 8 cards
DeepSeek V4 Flash 0731: more than 8 cards
MiMo V2.6 Flash: more than 8 cards
IQuest-Q1 320B-A15B: more than 8 cards
GLM-5.3 Flash: more than 8 cards
MiniMax M3: more than 8 cards
Ling 3.1 Flash 560B-A25B: more than 8 cards
DeepSeek V3 / R1: more than 8 cards
DeepSeek V3.2: more than 8 cards
GLM-5.3: more than 8 cards
GLM-5.2: more than 8 cards
DeepSeek V4.1 Flash 1017: more than 8 cards
DeepSeek V4.1 Flash: more than 8 cards
Hy4 Preview 770B-A49B: more than 8 cards
MiMo V2.6 Pro: more than 8 cards
DeepSeek V4 Pro: more than 8 cards
Qwen3.8 2.4T-A95B: more than 8 cards
Kimi K3: more than 8 cards

Best models for RTX 4070 12GB with room for long context

The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the card takes at that precision and the writing speed with that context in the cache, from 504 GB/s of bandwidth.

Model	Context	Memory	Left over	Longest context	Tokens/s
Gemma 4 12B 12.0B	32K	8.98 GB	3.02 GB	178K	32–45
ZDTaichu 5.0 9B 9.8B	32K	7.67 GB	4.33 GB	128K (full)	37–53
Ornith 1.5 9B 9.7B	32K	7.58 GB	4.42 GB	145K	38–53
Qwen3.5 9B 9.7B	32K	7.58 GB	4.42 GB	145K	38–53
LensVLM 9B 9.4B	32K	7.43 GB	4.57 GB	150K	39–54

llama-server commands for one RTX 4070 12GB

GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on the RTX 4070 12GB. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.

Gemma 4 12B, Q4_K_M with 178K tokens: 25–35 tokens/s on the RTX 4070 12GB

llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_M -c 182272 -np 1

gemma-4-12b-it-Q4_K_M.gguf, 7.1 GB: 11.5 GB used and 525 MB free of the RTX 4070 12GB's 12 GB, 25–35 tokens/s once the 178K cache is full. -np 1: one slot, one sliding window.

Ornith 1.5 9B, Q4_K_M with 143K tokens: 25–35 tokens/s on the RTX 4070 12GB

llama-server -hf bartowski/Ornith-1.5-9B-GGUF:Q4_K_M -c 146432

Ornith-1.5-9B-Q4_K_M.gguf, 5.9 GB: 11.5 GB used and 542 MB free of the RTX 4070 12GB's 12 GB, 25–35 tokens/s once the 143K cache is full.

Qwen3.5 9B, Q4_K_M with 145K tokens: 25–35 tokens/s on the RTX 4070 12GB

llama-server -hf unsloth/Qwen3.5-9B-GGUF:Q4_K_M -c 148480

Qwen3.5-9B-Q4_K_M.gguf, 5.7 GB: 11.5 GB used and 545 MB free of the RTX 4070 12GB's 12 GB, 25–35 tokens/s once the 145K cache is full.

How fast the RTX 4070 12GB writes as the context fills

Tokens per second at Q4_K_M for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the card holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 504 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside. With the same memory, RTX 3060 12GB, 360 GB/s, writes 28% slower on average; Arc B580 12GB, 456 GB/s, writes 9% slower on average; RTX 5070 12GB, 672 GB/s, writes 31% faster on average.

Model	8K	32K	Longest	At the longest	Q8_0 at 8K	Bandwidth ceiling at 8K	1,000-token reply at 32K
Gemma 4 12B	33–47	32–45	178K	25–35	Does not fit	64	22–31 s
ZDTaichu 5.0 9B	42–59	37–53	128K	26–36	25–35	81	19–27 s
Ornith 1.5 9B	42–60	38–53	145K	25–35	25–35	83	19–26 s
Qwen3.5 9B	42–60	38–53	145K	25–35	25–35	83	19–26 s
LensVLM 9B	43–61	39–54	150K	25–35	26–36	85	18–26 s
Ornith 1.0 9B	43–61	39–54	150K	25–35	26–36	85	18–26 s
MiMo V2.6 Distill Qwen 9B	43–61	39–54	150K	25–35	26–36	85	18–26 s
Bespoke Nimble 9B	45–64	—	8K	45–64	27–38	89	16–22 s
Granite 4.2 8B	39–55	25–35	32K	25–35	25–35	76	29–40 s
LFM2.5 8B-A1B	122–222	98–175	128,000	56–97	79–138	500	6–10 s

RTX 4070 12GB against the other 12 GB cards

The same models fit on every card with 12 GB, so speed is what separates them. By bandwidth the RTX 4070 12GB ranks 2 of 4 at 504 GB/s. Over the 12 biggest models that fit at Q4_K_M with 8K context, the RTX 3060 12GB (360 GB/s) writes 28% slower, the Arc B580 12GB (456 GB/s) writes 9% slower and the RTX 5070 12GB (672 GB/s) writes 31% faster than the RTX 4070 12GB. Each cell below is the other card's tokens per second minus this one's, midpoints of the estimated ranges.

Model	RTX 4070 12GB tokens/s	RTX 3060 12GB	Arc B580 12GB	RTX 5070 12GB
Gemma 4 12B	33–47	−11	−3.7	+13
ZDTaichu 5.0 9B	42–59	−14	−4.6	+16
Ornith 1.5 9B	42–60	−14	−4.7	+16
Qwen3.5 9B	42–60	−14	−4.7	+16
LensVLM 9B	43–61	−15	−4.8	+16
Ornith 1.0 9B	43–61	−15	−4.8	+16
MiMo V2.6 Distill Qwen 9B	43–61	−15	−4.8	+16
Bespoke Nimble 9B	45–64	−15	−5.0	+17
Granite 4.2 8B	39–55	−13	−4.3	+15
LFM2.5 8B-A1B	122–222	−44	−14	+47
Qwen3 8B	42–60	−14	−4.6	+16
Llama 3.1 8B	44–62	−15	−4.8	+17

What more cards or MoE offload unlock

One, two and four RTX 4070 12GB cards in one tensor-parallel group (vLLM or SGLang), with the models each step adds at Q4_K_M and 8K context and their tokens per second on that group.

Setup	Memory	Bandwidth	Models at Q4_K_M	Adds (tokens/s)
RTX 4070 12GB	12 GB	504 GB/s	20	This page
2× RTX 4070 12GB	24 GB	1,008 GB/s	38	gpt-oss-20b 83–155, Gemma 4 26B-A4B 76–142, Hemmingway-1 27B 29–42, ThinkingCap Qwen3.8 27B 29–41, Qwen3.6 27B 29–41, Qwen3.8 27B 29–41, Granite 4.2 30B 25–36, Muse Glimmer 30B 27–39, Qwen3-Coder 30B-A3B 78–146, Qwen3 30B-A3B 78–146, Xing 4.0 29B-A4B 78–146, GLM-4.7 Flash 91–173, Gemma 4 31B 24–35, Nemotron 3 Nano 30B-A3B 94–179, LLM-jp-4.1 32B-A3B Thinking 77–144, Ornith 1.0 35B 99–191, Ornith 1.5 35B-A3B 99–191, Qwen3.6 35B-A3B 99–191
4× RTX 4070 12GB	48 GB	2,016 GB/s	39	K2-Horizon MoVA 36B-A4B 98–189

Every mixture-of-experts GGUF file the --n-cpu-moe planner has measured, on one RTX 4070 12GB with 32K tokens of context (or the model's whole window): the smallest --n-cpu-moe that fits the card (0 means the whole file fits), the memory on the card and in system RAM, and tokens per second from 504 GB/s of card bandwidth with DDR5-5600, 2 channels (89.6 GB/s) or DDR4-3200, 2 channels (51.2 GB/s). 11 of these models do not fit whole and need offload.

Model	GGUF	--n-cpu-moe	On the card	In system RAM	Tokens/s, DDR5	Tokens/s, DDR4
gpt-oss-20b	MXFP4	2	11.7 GB	1.36 GB	37–63	33–56
Gemma 4 26B-A4B	UD-Q4_K_M	12	11.6 GB	6.05 GB	27–46	21–36
Qwen3 30B-A3B	Q4_K_M	27	11.9 GB	9.33 GB	18–31	14–24
Qwen3 30B-A3B	Q8_0	37	11.8 GB	22.4 GB	11–19	8–13
GLM-4.7 Flash	UD-Q4_K_XL	23	11.7 GB	7.24 GB	23–39	18–30
GLM-4.7 Flash	Q4_K_M	24	11.8 GB	7.93 GB	22–38	17–28
Nemotron 3 Nano 30B-A3B	Q4_K_M	30	11.9 GB	12.2 GB	26–44	18–31
Qwen3.6 35B-A3B	UD-IQ4_XS	17	11.7 GB	6.41 GB	35–60	29–50
Ornith 1.5 35B-A3B	Q4_K_M	22	11.6 GB	10.2 GB	34–58	26–44
Qwen3.6 35B-A3B	UD-Q4_K_M	22	11.8 GB	10.5 GB	30–51	23–39
Qwen3.6 35B-A3B	Q8_0	30	11.6 GB	24.4 GB	19–33	14–23
Ornith 1.5 35B-A3B	Q8_0	31	11.6 GB	25.2 GB	19–32	13–22
Qwen3-Coder-Next	IQ4_XS	37	11.8 GB	29.6 GB	25–42	17–29
Qwen3-Coder-Next	Q4_K_M	39	11.7 GB	35.3 GB	22–37	15–25
Qwen3-Coder-Next	UD-Q4_K_XL	40	11.4 GB	36.6 GB	19–33	14–23
gpt-oss-120b	MXFP4	31	11.6 GB	49.6 GB	12–20	8–13
Qwen3.5 122B-A10B	Q4_K_M	45	11.1 GB	61.9 GB	8–14	6–9
Qwen3.8 Flash Next	UD-Q2_K_XL	41	11.3 GB	37.1 GB	17–28	12–21
Qwen3.8 Flash Next	UD-IQ3_XXS	42	11.1 GB	40.1 GB	15–26	11–19
Qwen3.8 Flash Next	GSQ-RCO IQ3_S	43	11.2 GB	41.6 GB	15–25	11–18
Qwen3.8 Flash Next	UD-Q3_K_XL	44	11.0 GB	47.7 GB	13–22	10–16
Qwen3.8 Flash Next	UD-IQ4_XS	44	11.3 GB	50.8 GB	13–22	9–16
Qwen3.8 Flash Next	UD-Q4_K_XL	45	11.1 GB	67.5 GB	11–18	8–13
Qwen3.8 Flash Next	UD-Q5_K_XL	45	12.0 GB	86.5 GB	9–16	6–11

In system RAM counts the offloaded experts plus the input embedding (token_embd), which llama.cpp keeps in RAM in any case.

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Nearby GPUs and setups

How many of the tracked models each one holds at Q4_K_M with 8K context, against 20 here.

Setup	How it relates	Memory	Bandwidth	Models at Q4_K_M
RTX 3060 12GB	Same memory	12 GB	360 GB/s	20 (same)
Arc B580 12GB	Same memory	12 GB	456 GB/s	20 (same)
RTX 5070 12GB	Same memory	12 GB	672 GB/s	20 (same)
RTX 4060 8GB	Next size down	8 GB	272 GB/s	19 (−1)
RTX 4060 Ti 16GB	Next size up	16 GB	288 GB/s	21 (+1)

Model numbers read from Hugging Face on 2026-10-20.