ModelVRAM
EN 中文

What LLMs can an RTX 4060 8GB run?

An RTX 4060 8GB runs 19 of the 69 open models tracked here at Q4_K_M with 8K context; the largest is ZDTaichu 5.0 9B (9.8B parameters, 6.85 GB). It reads memory at 272 GB/s. 1 more fits at a lower precision. Each fit leaves at least 0.5 GB free on the RTX 4060 8GB (why a card does not give a model all its memory).

Not sure your PC has an RTX 4060 8GB? Detect your GPU in the browser.

Models that fit on one RTX 4060 8GB

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first.

Model Best precision Memory Longest context Tokens/s
Gemma 4 12B 12.0B GGUF Q3_K_M 7.14 GB 28K 22–31
ZDTaichu 5.0 9B 9.8B GGUF Q4_K_M 6.85 GB 26K 23–32
Ornith 1.5 9B 9.7B GGUF Q4_K_M 6.76 GB 29K 24–33
Qwen3.5 9B 9.7B GGUF Q4_K_M 6.76 GB 29K 24–33
LensVLM 9B 9.4B GGUF Q4_K_M 6.61 GB 33K 24–34
Ornith 1.0 9B 9.4B GGUF Q4_K_M 6.61 GB 33K 24–34
MiMo V2.6 Distill Qwen 9B 9.4B GGUF Q4_K_M 6.61 GB 33K 24–34
Bespoke Nimble 9B 9.0B GGUF Q5_K_M 7.28 GB 8K (full) 22–30
Granite 4.2 8B 8.8B GGUF Q4_K_M 7.32 GB 8K 22–30
LFM2.5 8B-A1B 8.5B, 1.5B active GGUF Q5_K_M 6.75 GB 65K 63–110
Qwen3 8B 8.2B GGUF Q4_K_M 6.81 GB 12K 23–33
Llama 3.1 8B 8.0B GGUF Q5_K_M 7.43 GB 8K 21–30
Gemma 4 E4B 8.0B GGUF Q6_K 7.40 GB 13K 22–30
Ling 3.0 Tiny 7.9B-A1.3B 7.9B, 1.3B active GGUF Q6_K 7.19 GB 50K 66–114
LightOnOCR-3 4B 4.5B GGUF Q8_0 5.72 GB 59K 28–39
Spark-X2.5 4B 4.1B GGUF Q8_0 5.40 GB 62K 30–42
Nemotron 3 Nano 4B 4.0B GGUF Q8_0 4.96 GB 155K 33–46
Granite 4.2 3B 3.7B GGUF Q8_0 5.17 GB 34K 31–44
MiniCPM5 2B 2.5B FP16 / BF16 6.02 GB 40K 27–37
Limite 1B Violetto 1.0B FP16 / BF16 2.79 GB 128K (full) 61–87

Too big for one RTX 4060 8GB at 8K context

How many RTX 4060 8GB cards these need at Q4_K_M (gpt-oss-20b and gpt-oss-120b at MXFP4) with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.

Best models for RTX 4060 8GB with room for long context

The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the card takes at that precision and the writing speed with that context in the cache, from 272 GB/s of bandwidth.

ModelContextMemoryLeft overLongest contextTokens/s
LensVLM 9B 9.4B 32K 7.43 GB 581 MB 33K 21–30
Ornith 1.0 9B 9.4B 32K 7.43 GB 581 MB 33K 21–30
MiMo V2.6 Distill Qwen 9B 9.4B 32K 7.43 GB 581 MB 33K 21–30
Bespoke Nimble 9B 9.0B 8K 6.32 GB 1.68 GB 8K (full) 25–35
LFM2.5 8B-A1B 8.5B 32K 6.16 GB 1.84 GB 128,000 (full) 57–99

llama-server commands for one RTX 4060 8GB

GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on the RTX 4060 8GB. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.

LensVLM 9B, Q4_K_M with 29K tokens: 22–30 tokens/s on the RTX 4060 8GB

llama-server -hf bartowski/LensVLM-9B-GGUF:Q4_K_M -c 29696

LensVLM-9B-Q4_K_M.gguf, 5.8 GB: 7.48 GB used and 532 MB free of the RTX 4060 8GB's 8 GB, 22–30 tokens/s once the 29K cache is full.

Ornith 1.0 9B, Q4_K_M with 27K tokens: 22–30 tokens/s on the RTX 4060 8GB

llama-server -hf bartowski/deepreinforce-ai_Ornith-1.0-9B-GGUF:Q4_K_M -c 27648

deepreinforce-ai_Ornith-1.0-9B-Q4_K_M.gguf, 5.9 GB: 7.48 GB used and 529 MB free of the RTX 4060 8GB's 8 GB, 22–30 tokens/s once the 27K cache is full.

MiMo V2.6 Distill Qwen 9B, Q4_K_M with 29K tokens: 22–30 tokens/s on the RTX 4060 8GB

llama-server -hf bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q4_K_M -c 29696

MiMo-V2.6-Distill-Qwen-9B-Q4_K_M.gguf, 5.8 GB: 7.48 GB used and 532 MB free of the RTX 4060 8GB's 8 GB, 22–30 tokens/s once the 29K cache is full.

How fast the RTX 4060 8GB writes as the context fills

Tokens per second at Q4_K_M for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the card holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 272 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside.

Model8K32KLongestAt the longestQ8_0 at 8KBandwidth ceiling at 8K1,000-token reply at 32K
LensVLM 9B 24–34 21–30 33K 21–30 Does not fit 46 34–47 s
Ornith 1.0 9B 24–34 21–30 33K 21–30 Does not fit 46 34–47 s
MiMo V2.6 Distill Qwen 9B 24–34 21–30 33K 21–30 Does not fit 46 34–47 s
Bespoke Nimble 9B 25–35 — 8K 25–35 Does not fit 48 28–40 s
LFM2.5 8B-A1B 72–126 57–99 128,000 31–53 Does not fit 270 10–18 s
Gemma 4 E4B 29–40 27–37 116K 21–29 Does not fit 54 27–38 s
Ling 3.0 Tiny 7.9B-A1.3B 85–149 72–126 128K 45–77 Does not fit 323 8–14 s
LightOnOCR-3 4B 46–65 37–52 121K 21–30 28–39 90 19–27 s
Spark-X2.5 4B 48–68 37–52 111K 21–30 30–42 94 19–27 s
Nemotron 3 Nano 4B 54–77 47–67 256K 22–30 33–46 107 15–21 s

Near misses on RTX 4060 8GB

Models that miss at Q4_K_M with 32,768 tokens of context (or their whole window), or leave under 0.5 GB free there, but fit with a shorter context or 3-bit weights, or are short by at most a quarter of the memory and fit with MoE offload. Smallest shortfall first.

ModelShort by at Q4_K_MFits instead
Ornith 1.5 9B 9.7B Tight, 427 MB free Q4_K_M with 29K context
Qwen3.5 9B 9.7B Tight, 427 MB free Q4_K_M with 29K context
ZDTaichu 5.0 9B 9.8B Tight, 338 MB free Q4_K_M with 26K context
Gemma 4 12B 12.0B 1,002 MB GGUF IQ3_XXS with 32K, 6.62 GB
Llama 3.1 8B 8.0B 1.88 GB Q4_K_M with 14K context

What more cards or MoE offload unlock

One, two and four RTX 4060 8GB cards in one tensor-parallel group (vLLM or SGLang), with the models each step adds at Q4_K_M and 8K context and their tokens per second on that group.

Setup Memory Bandwidth Models at Q4_K_M Adds (tokens/s)
RTX 4060 8GB 8 GB 272 GB/s 19 This page
2× RTX 4060 8GB 16 GB 544 GB/s 21 Gemma 4 12B 34–49, gpt-oss-20b 52–92
4× RTX 4060 8GB 32 GB 1,088 GB/s 39 Gemma 4 26B-A4B 81–151, Hemmingway-1 27B 31–45, ThinkingCap Qwen3.8 27B 31–44, Qwen3.6 27B 31–44, Qwen3.8 27B 31–44, Granite 4.2 30B 27–39, Muse Glimmer 30B 29–42, Qwen3-Coder 30B-A3B 83–155, Qwen3 30B-A3B 83–155, Xing 4.0 29B-A4B 83–155, GLM-4.7 Flash 96–183, Gemma 4 31B 26–37, Nemotron 3 Nano 30B-A3B 99–189, LLM-jp-4.1 32B-A3B Thinking 82–153, Ornith 1.0 35B 104–202, Ornith 1.5 35B-A3B 104–202, Qwen3.6 35B-A3B 104–202, K2-Horizon MoVA 36B-A4B 63–115

Every mixture-of-experts GGUF file the --n-cpu-moe planner has measured, on one RTX 4060 8GB with 32K tokens of context (or the model's whole window): the smallest --n-cpu-moe that fits the card (0 means the whole file fits), the memory on the card and in system RAM, and tokens per second from 272 GB/s of card bandwidth with DDR5-5600, 2 channels (89.6 GB/s) or DDR4-3200, 2 channels (51.2 GB/s). 11 of these models do not fit whole and need offload.

Model GGUF --n-cpu-moe On the card In system RAM Tokens/s, DDR5 Tokens/s, DDR4
gpt-oss-20b MXFP4 12 7.72 GB 5.31 GB 17–29 13–22
Gemma 4 26B-A4B UD-Q4_K_M 21 7.65 GB 10.0 GB 16–27 13–21
Qwen3 30B-A3B Q4_K_M 39 7.95 GB 13.3 GB 12–19 9–15
Qwen3 30B-A3B Q8_0 44 7.64 GB 26.6 GB 8–13 6–10
GLM-4.7 Flash UD-Q4_K_XL 35 7.90 GB 11.1 GB 14–24 11–18
GLM-4.7 Flash Q4_K_M 36 7.78 GB 11.9 GB 14–23 11–18
Nemotron 3 Nano 30B-A3B Q4_K_M 41 7.56 GB 16.5 GB 18–30 13–21
Qwen3.6 35B-A3B UD-IQ4_XS 28 7.89 GB 10.2 GB 21–35 17–29
Ornith 1.5 35B-A3B Q4_K_M 31 7.65 GB 14.2 GB 22–37 17–29
Qwen3.6 35B-A3B UD-Q4_K_M 31 7.68 GB 14.6 GB 19–31 15–25
Qwen3.6 35B-A3B Q8_0 35 7.59 GB 28.4 GB 14–23 10–17
Ornith 1.5 35B-A3B Q8_0 36 7.63 GB 29.2 GB 14–23 10–17
Qwen3-Coder-Next IQ4_XS 42 7.86 GB 33.6 GB 18–30 13–22
Qwen3-Coder-Next Q4_K_M 44 7.07 GB 39.9 GB 16–27 12–20
Qwen3-Coder-Next UD-Q4_K_XL 44 7.76 GB 40.2 GB 14–23 10–18
gpt-oss-120b MXFP4 34 6.88 GB 54.3 GB 9–16 6–11
Qwen3.5 122B-A10B Q4_K_M 48 7.02 GB 66.0 GB 6–10 4–7
Qwen3.8 Flash Next UD-Q2_K_XL 45 7.72 GB 40.7 GB 12–19 9–15
Qwen3.8 Flash Next UD-IQ3_XXS 46 7.36 GB 43.9 GB 11–18 8–14
Qwen3.8 Flash Next GSQ-RCO IQ3_S 46 7.95 GB 44.8 GB 10–17 8–14
Qwen3.8 Flash Next UD-Q3_K_XL 47 7.53 GB 51.2 GB 9–15 7–12
Qwen3.8 Flash Next UD-IQ4_XS 47 7.61 GB 54.6 GB 9–15 7–12
Qwen3.8 Flash Next UD-Q4_K_XL 47 7.96 GB 70.6 GB 8–13 6–10
Qwen3.8 Flash Next UD-Q5_K_XL 48 6.25 GB 92.2 GB 7–12 5–9

In system RAM counts the offloaded experts plus the input embedding (token_embd), which llama.cpp keeps in RAM in any case.

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Nearby GPUs and setups

How many of the tracked models each one holds at Q4_K_M with 8K context, against 19 here.

SetupHow it relatesMemoryBandwidthModels at Q4_K_M
RTX 3060 12GB Next size up 12 GB 360 GB/s 20 (+1)

Model numbers read from Hugging Face on .