What LLMs can an RTX 4060 Ti 16GB run?
An RTX 4060 Ti 16GB runs 21 of the 69 open models tracked here at Q4_K_M with 8K context; the largest is gpt-oss-20b (20.9B parameters, 14.8 GB at MXFP4). It reads memory at 288 GB/s. 13 more fit at a lower precision. Each fit leaves at least 0.5 GB free on the RTX 4060 Ti 16GB (why a card does not give a model all its memory).
Not sure your PC has an RTX 4060 Ti 16GB? Detect your GPU in the browser.
Can I run it on an RTX 4060 Ti 16GB with 32K context?
- Gemma 4 26B-A4B: Only Q3_K_M
- Gemma 4 31B: No, 2 cards
- Qwen3.8 27B: Only Q2_K
- gpt-oss-20b: Yes, MXFP4
- Llama 3.1 8B: Yes, up to Q8_0
- gpt-oss-120b: --n-cpu-moe 29
- Qwen3.6 35B-A3B: --n-cpu-moe 13
- Qwen3.6 27B: Only Q2_K
- Gemma 4 12B: Yes, up to Q8_0
- GLM-4.7 Flash: --n-cpu-moe 12
- Qwen3.8 Flash Next: --n-cpu-moe 42
- Nemotron 3 Nano 30B-A3B: Only Q2_K
- Qwen3-Coder-Next: --n-cpu-moe 35
- Llama 3.1 70B: No, 4 cards
Models that fit on one RTX 4060 Ti 16GB
The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first. gpt-oss-20b (14.8 GB, 32–54 tokens/s on the RTX 4060 Ti 16GB) is listed at MXFP4 only, as published.
| Model | Best precision | Memory | Longest context | Tokens/s |
|---|---|---|---|---|
| LLM-jp-4.1 32B-A3B Thinking 32.1B, 3.8B active | GGUF Q2_K | 14.8 GB | 17K | 38–65 |
| Nemotron 3 Nano 30B-A3B 31.6B, 3.5B active | GGUF Q2_K | 14.1 GB | 225K | 53–91 |
| GLM-4.7 Flash 31.2B, 3B active | GGUF Q2_K | 14.3 GB | 28K | 47–81 |
| Xing 4.0 29B-A4B 31.2B, 4B active | GGUF Q2_K | 14.3 GB | 33K | 40–68 |
| Qwen3-Coder 30B-A3B 30.5B, 3.3B active | GGUF Q2_K | 14.4 GB | 18K | 37–64 |
| Qwen3 30B-A3B 30.5B, 3.3B active | GGUF Q2_K | 14.4 GB | 18K | 37–64 |
| Muse Glimmer 30B 29.8B | GGUF Q2_K | 13.5 GB | 128K (full) | 12–17 |
| Granite 4.2 30B 29.3B | GGUF Q2_K | 15.3 GB | 8K | 11–15 |
| ThinkingCap Qwen3.8 27B 27.8B | GGUF Q3_K_M | 15.0 GB | 15K | 11–15 |
| Qwen3.6 27B 27.8B | GGUF Q3_K_M | 15.0 GB | 15K | 11–15 |
| Qwen3.8 27B 27.8B | GGUF Q3_K_M | 15.0 GB | 15K | 11–15 |
| Hemmingway-1 27B 27.3B | GGUF Q3_K_M | 14.7 GB | 19K | 11–15 |
| Gemma 4 26B-A4B 25.8B, 4B active | GGUF Q3_K_M | 13.9 GB | 81K | 34–57 |
| gpt-oss-20b 20.9B, 3.6B active | As published (MXFP4) | 14.8 GB | 34K | 32–54 |
| Gemma 4 12B 12.0B | GGUF Q8_0 | 14.2 GB | 85K | 12–16 |
| ZDTaichu 5.0 9B 9.8B | GGUF Q8_0 | 11.4 GB | 126K | 15–20 |
| Ornith 1.5 9B 9.7B | GGUF Q8_0 | 11.3 GB | 130K | 15–20 |
| Qwen3.5 9B 9.7B | GGUF Q8_0 | 11.3 GB | 130K | 15–20 |
| LensVLM 9B 9.4B | GGUF Q8_0 | 11.0 GB | 138K | 15–21 |
| Ornith 1.0 9B 9.4B | GGUF Q8_0 | 11.0 GB | 138K | 15–21 |
| MiMo V2.6 Distill Qwen 9B 9.4B | GGUF Q8_0 | 11.0 GB | 138K | 15–21 |
| Bespoke Nimble 9B 9.0B | GGUF Q8_0 | 10.5 GB | 8K (full) | 16–22 |
| Granite 4.2 8B 8.8B | GGUF Q8_0 | 11.4 GB | 31K | 15–20 |
| LFM2.5 8B-A1B 8.5B, 1.5B active | GGUF Q8_0 | 9.82 GB | 128,000 (full) | 47–82 |
| Qwen3 8B 8.2B | GGUF Q8_0 | 10.7 GB | 39K | 16–22 |
| Llama 3.1 8B 8.0B | GGUF Q8_0 | 10.3 GB | 45K | 16–22 |
| Gemma 4 E4B 8.0B | GGUF Q8_0 | 9.38 GB | 128K (full) | 18–25 |
| Ling 3.0 Tiny 7.9B-A1.3B 7.9B, 1.3B active | GGUF Q8_0 | 9.15 GB | 128K (full) | 55–95 |
| LightOnOCR-3 4B 4.5B | FP16 / BF16 | 10.1 GB | 165K | 17–23 |
| Spark-X2.5 4B 4.1B | FP16 / BF16 | 9.35 GB | 166K | 18–25 |
| Nemotron 3 Nano 4B 4.0B | FP16 / BF16 | 8.78 GB | 256K (full) | 19–26 |
| Granite 4.2 3B 3.7B | FP16 / BF16 | 8.69 GB | 87K | 19–27 |
| MiniCPM5 2B 2.5B | FP16 / BF16 | 6.02 GB | 128K (full) | 28–39 |
| Limite 1B Violetto 1.0B | FP16 / BF16 | 2.79 GB | 128K (full) | 64–92 |
Too big for one RTX 4060 Ti 16GB at 8K context
How many RTX 4060 Ti 16GB cards these need at Q4_K_M (gpt-oss-120b at MXFP4) with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.
- Gemma 4 31B: 2 cards
- Ornith 1.0 35B: 2 cards
- Ornith 1.5 35B-A3B: 2 cards
- Qwen3.6 35B-A3B: 2 cards
- K2-Horizon MoVA 36B-A4B: 2 cards
- Llama 3.1 70B: 4 cards
- Kolibri-1: 4 cards
- Qwen3-Coder-Next: 4 cards
- AliceAI Foundation 80B-A3B: 4 cards
- gpt-oss-120b: 8 cards
- Nemotron 3 Super 120B-A12B: 8 cards
- Nemotron 3.5 Super VL 120B-A12B: 8 cards
- Qwen3.5 122B-A10B: 8 cards
- Mistral Medium 3.5 128B: 8 cards
- Qwen3.8 Flash Next: 8 cards
- Step 3.7 Flash 196B-A11B: more than 8 cards
- MiniMax M2.7: more than 8 cards
- DeepSeek V4 Flash: more than 8 cards
- DeepSeek V4 Flash 0731: more than 8 cards
- MiMo V2.6 Flash: more than 8 cards
- IQuest-Q1 320B-A15B: more than 8 cards
- GLM-5.3 Flash: more than 8 cards
- MiniMax M3: more than 8 cards
- Ling 3.1 Flash 560B-A25B: more than 8 cards
- DeepSeek V3 / R1: more than 8 cards
- DeepSeek V3.2: more than 8 cards
- GLM-5.3: more than 8 cards
- GLM-5.2: more than 8 cards
- DeepSeek V4.1 Flash 1017: more than 8 cards
- DeepSeek V4.1 Flash: more than 8 cards
- Hy4 Preview 770B-A49B: more than 8 cards
- MiMo V2.6 Pro: more than 8 cards
- DeepSeek V4 Pro: more than 8 cards
- Qwen3.8 2.4T-A95B: more than 8 cards
- Kimi K3: more than 8 cards
Best models for RTX 4060 Ti 16GB with room for long context
The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the card takes at that precision and the writing speed with that context in the cache, from 288 GB/s of bandwidth. gpt-oss-20b is at MXFP4, as published.
| Model | Context | Memory | Left over | Longest context | Tokens/s |
|---|---|---|---|---|---|
| gpt-oss-20b 20.9B, MXFP4 | 32K | 15.4 GB | 571 MB | 34K | 26–44 |
| Gemma 4 12B 12.0B | 32K | 8.98 GB | 7.02 GB | 256K (full) | 19–26 |
| ZDTaichu 5.0 9B 9.8B | 32K | 7.67 GB | 8.33 GB | 128K (full) | 22–30 |
| Ornith 1.5 9B 9.7B | 32K | 7.58 GB | 8.42 GB | 256K (full) | 22–31 |
| Qwen3.5 9B 9.7B | 32K | 7.58 GB | 8.42 GB | 256K (full) | 22–31 |
llama-server commands for one RTX 4060 Ti 16GB
GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on the RTX 4060 Ti 16GB. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.
gpt-oss-20b, MXFP4 with 34K tokens: 26–43 tokens/s on the RTX 4060 Ti 16GB
llama-server -hf ggml-org/gpt-oss-20b-GGUF:MXFP4 -c 34816 -np 1gpt-oss-20b-MXFP4.gguf, 12.1 GB: 15.5 GB used and 518 MB free of the RTX 4060 Ti 16GB's 16 GB, 26–43 tokens/s once the 34K cache is full. -np 1: one slot, one sliding window.
Gemma 4 12B, Q4_K_M with 256K tokens: 13–18 tokens/s on the RTX 4060 Ti 16GB
llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_M -c 262144 -np 1gemma-4-12b-it-Q4_K_M.gguf, 7.1 GB: 12.8 GB used and 3.17 GB free of the RTX 4060 Ti 16GB's 16 GB, 13–18 tokens/s once the 256K cache is full. -np 1: one slot, one sliding window.
How fast the RTX 4060 Ti 16GB writes as the context fills
Tokens per second at Q4_K_M (gpt-oss-20b at MXFP4) for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the card holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 288 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside. With the same memory, RTX 5060 Ti 16GB, 448 GB/s, writes 53% faster on average; RX 9070 XT 16GB, 644 GB/s, writes 115% faster on average; RTX 4070 Ti Super 16GB, 672 GB/s, writes 124% faster on average; RTX 4080 Super 16GB, 736 GB/s, writes 144% faster on average; RTX 5070 Ti 16GB, 896 GB/s, writes 192% faster on average; RTX 5080 16GB, 960 GB/s, writes 211% faster on average.
| Model | 8K | 32K | Longest | At the longest | Q8_0 at 8K | Bandwidth ceiling at 8K | 1,000-token reply at 32K |
|---|---|---|---|---|---|---|---|
| gpt-oss-20b | 32–54 | 26–44 | 34K | 26–43 | MXFP4 only | 111 | 23–38 s |
| Gemma 4 12B | 20–27 | 19–26 | 256K | 13–18 | 12–16 | 37 | 39–54 s |
| ZDTaichu 5.0 9B | 25–34 | 22–30 | 128K | 15–21 | 15–20 | 46 | 33–46 s |
| Ornith 1.5 9B | 25–35 | 22–31 | 256K | 11–15 | 15–20 | 47 | 33–45 s |
| Qwen3.5 9B | 25–35 | 22–31 | 256K | 11–15 | 15–20 | 47 | 33–45 s |
| LensVLM 9B | 26–36 | 23–31 | 256K | 11–15 | 15–21 | 48 | 32–44 s |
| Ornith 1.0 9B | 26–36 | 23–31 | 256K | 11–15 | 15–21 | 48 | 32–44 s |
| MiMo V2.6 Distill Qwen 9B | 26–36 | 23–31 | 256K | 11–15 | 15–21 | 48 | 32–44 s |
| Bespoke Nimble 9B | 27–37 | — | 8K | 27–37 | 16–22 | 51 | 27–37 s |
| Granite 4.2 8B | 23–32 | 14–20 | 55K | 11–15 | 15–20 | 43 | 50–69 s |
RTX 4060 Ti 16GB against the other 16 GB cards
The same models fit on every card with 16 GB, so speed is what separates them. By bandwidth the RTX 4060 Ti 16GB is the slowest of the 7 cards with 16 GB at 288 GB/s. Over the 12 biggest models that fit at Q4_K_M with 8K context, the RTX 5060 Ti 16GB (448 GB/s) writes 53% faster, the RX 9070 XT 16GB (644 GB/s) writes 115% faster, the RTX 4070 Ti Super 16GB (672 GB/s) writes 124% faster, the RTX 4080 Super 16GB (736 GB/s) writes 144% faster, the RTX 5070 Ti 16GB (896 GB/s) writes 192% faster and the RTX 5080 16GB (960 GB/s) writes 211% faster than the RTX 4060 Ti 16GB. Each cell below is the other card's tokens per second minus this one's, midpoints of the estimated ranges.
| Model | RTX 4060 Ti 16GB tokens/s | RTX 5060 Ti 16GB | RX 9070 XT 16GB | RTX 4070 Ti Super 16GB | RTX 4080 Super 16GB | RTX 5070 Ti 16GB | RTX 5080 16GB |
|---|---|---|---|---|---|---|---|
| gpt-oss-20b | 32–54 | +23 | +49 | +53 | +61 | +82 | +90 |
| Gemma 4 12B | 20–27 | +13 | +28 | +30 | +34 | +46 | +51 |
| ZDTaichu 5.0 9B | 25–34 | +16 | +34 | +37 | +43 | +58 | +63 |
| Ornith 1.5 9B | 25–35 | +16 | +35 | +38 | +44 | +58 | +64 |
| Qwen3.5 9B | 25–35 | +16 | +35 | +38 | +44 | +58 | +64 |
| LensVLM 9B | 26–36 | +16 | +36 | +38 | +44 | +60 | +65 |
| Ornith 1.0 9B | 26–36 | +16 | +36 | +38 | +44 | +60 | +65 |
| MiMo V2.6 Distill Qwen 9B | 26–36 | +16 | +36 | +38 | +44 | +60 | +65 |
| Bespoke Nimble 9B | 27–37 | +17 | +37 | +40 | +46 | +62 | +68 |
| Granite 4.2 8B | 23–32 | +15 | +32 | +35 | +40 | +54 | +59 |
| LFM2.5 8B-A1B | 76–133 | +51 | +108 | +115 | +132 | +172 | +188 |
| Qwen3 8B | 25–34 | +16 | +35 | +37 | +43 | +58 | +64 |
Near misses on RTX 4060 Ti 16GB
Models that miss at Q4_K_M with 32,768 tokens of context (or their whole window), or leave under 0.5 GB free there, but fit with a shorter context or 3-bit weights, or are short by at most a quarter of the memory and fit with MoE offload. Smallest shortfall first.
| Model | Short by at Q4_K_M | Fits instead |
|---|---|---|
| Gemma 4 26B-A4B 25.8B | 1.50 GB | GGUF Q3_K_M with 32K, 14.4 GB |
| Muse Glimmer 30B 29.8B | 3.51 GB | GGUF IQ3_XXS with 32K, 13.6 GB |
| Hemmingway-1 27B 27.3B | 3.63 GB | GGUF IQ3_XXS with 32K, 14.2 GB |
| ThinkingCap Qwen3.8 27B 27.8B | 3.92 GB | GGUF IQ3_XXS with 32K, 14.4 GB |
| Qwen3.6 27B 27.8B | 3.92 GB | GGUF IQ3_XXS with 32K, 14.4 GB |
What more cards or MoE offload unlock
One, two and four RTX 4060 Ti 16GB cards in one tensor-parallel group (vLLM or SGLang), with the models each step adds at Q4_K_M and 8K context and their tokens per second on that group.
| Setup | Memory | Bandwidth | Models at Q4_K_M | Adds (tokens/s) |
|---|---|---|---|---|
| RTX 4060 Ti 16GB | 16 GB | 288 GB/s | 21 | This page |
| 2× RTX 4060 Ti 16GB | 32 GB | 576 GB/s | 39 | Gemma 4 26B-A4B 49–88, Hemmingway-1 27B 17–25, ThinkingCap Qwen3.8 27B 17–24, Qwen3.6 27B 17–24, Qwen3.8 27B 17–24, Granite 4.2 30B 15–21, Muse Glimmer 30B 16–23, Qwen3-Coder 30B-A3B 51–91, Qwen3 30B-A3B 51–91, Xing 4.0 29B-A4B 51–91, GLM-4.7 Flash 60–109, Gemma 4 31B 14–20, Nemotron 3 Nano 30B-A3B 62–113, LLM-jp-4.1 32B-A3B Thinking 50–89, Ornith 1.0 35B 67–122, Ornith 1.5 35B-A3B 67–122, Qwen3.6 35B-A3B 67–122, K2-Horizon MoVA 36B-A4B 37–65 |
| 4× RTX 4060 Ti 16GB | 64 GB | 1,152 GB/s | 43 | Llama 3.1 70B 13–19, Kolibri-1 97–185, Qwen3-Coder-Next 107–208, AliceAI Foundation 80B-A3B 107–208 |
Every mixture-of-experts GGUF file the --n-cpu-moe planner has measured, on one RTX 4060 Ti 16GB with 32K tokens of
context (or the model's whole window): the smallest --n-cpu-moe that fits the card (0 means the whole file fits),
the memory on the card and in system RAM, and tokens per second from 288 GB/s of card
bandwidth with DDR5-5600, 2 channels (89.6 GB/s) or DDR4-3200, 2 channels (51.2 GB/s).
10 of these models do not fit whole and need offload.
| Model | GGUF | --n-cpu-moe | On the card | In system RAM | Tokens/s, DDR5 | Tokens/s, DDR4 |
|---|---|---|---|---|---|---|
| gpt-oss-20b | MXFP4 | 0 | 12.5 GB | 587 MB | 24–41 | 24–41 |
| Gemma 4 26B-A4B | UD-Q4_K_M | 3 | 15.6 GB | 2.06 GB | 22–36 | 20–35 |
| Qwen3 30B-A3B | Q4_K_M | 15 | 15.9 GB | 5.34 GB | 14–24 | 13–21 |
| Qwen3 30B-A3B | Q8_0 | 31 | 15.4 GB | 18.8 GB | 9–15 | 7–12 |
| GLM-4.7 Flash | UD-Q4_K_XL | 10 | 15.8 GB | 3.13 GB | 18–31 | 16–28 |
| GLM-4.7 Flash | Q4_K_M | 12 | 15.8 GB | 3.94 GB | 18–30 | 16–27 |
| Nemotron 3 Nano 30B-A3B | Q4_K_M | 21 | 15.4 GB | 8.70 GB | 22–38 | 18–30 |
| Qwen3.6 35B-A3B | UD-IQ4_XS | 5 | 15.9 GB | 2.24 GB | 25–43 | 24–41 |
| Ornith 1.5 35B-A3B | Q4_K_M | 13 | 15.6 GB | 6.20 GB | 27–45 | 23–39 |
| Qwen3.6 35B-A3B | UD-Q4_K_M | 13 | 15.8 GB | 6.39 GB | 22–38 | 20–34 |
| Qwen3.6 35B-A3B | Q8_0 | 25 | 15.6 GB | 20.4 GB | 16–27 | 12–20 |
| Ornith 1.5 35B-A3B | Q8_0 | 26 | 15.6 GB | 21.2 GB | 15–26 | 12–20 |
| Qwen3-Coder-Next | IQ4_XS | 32 | 15.8 GB | 25.7 GB | 20–34 | 15–26 |
| Qwen3-Coder-Next | Q4_K_M | 35 | 15.3 GB | 31.6 GB | 18–30 | 13–23 |
| Qwen3-Coder-Next | UD-Q4_K_XL | 35 | 15.9 GB | 32.0 GB | 15–26 | 12–20 |
| gpt-oss-120b | MXFP4 | 29 | 14.8 GB | 46.4 GB | 10–17 | 7–12 |
| Qwen3.5 122B-A10B | Q4_K_M | 42 | 15.2 GB | 57.8 GB | 7–11 | 5–8 |
| Qwen3.8 Flash Next | UD-Q2_K_XL | 36 | 15.7 GB | 32.6 GB | 13–21 | 10–17 |
| Qwen3.8 Flash Next | UD-IQ3_XXS | 37 | 15.8 GB | 35.4 GB | 12–19 | 9–16 |
| Qwen3.8 Flash Next | GSQ-RCO IQ3_S | 39 | 15.3 GB | 37.5 GB | 11–19 | 9–15 |
| Qwen3.8 Flash Next | UD-Q3_K_XL | 40 | 15.2 GB | 43.5 GB | 10–17 | 8–13 |
| Qwen3.8 Flash Next | UD-IQ4_XS | 40 | 15.8 GB | 46.4 GB | 10–16 | 8–13 |
| Qwen3.8 Flash Next | UD-Q4_K_XL | 42 | 15.5 GB | 63.1 GB | 8–14 | 6–11 |
| Qwen3.8 Flash Next | UD-Q5_K_XL | 43 | 15.8 GB | 82.7 GB | 8–13 | 6–9 |
In system RAM counts the offloaded experts plus the input embedding (token_embd), which llama.cpp keeps in RAM in any case.
How these numbers are worked out
Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.
Nearby GPUs and setups
How many of the tracked models each one holds at Q4_K_M with 8K context, against 21 here.
| Setup | How it relates | Memory | Bandwidth | Models at Q4_K_M |
|---|---|---|---|---|
| RTX 5060 Ti 16GB | Same memory | 16 GB | 448 GB/s | 21 (same) |
| RX 9070 XT 16GB | Same memory | 16 GB | 644 GB/s | 21 (same) |
| RTX 4070 Ti Super 16GB | Same memory | 16 GB | 672 GB/s | 21 (same) |
| RTX 4080 Super 16GB | Same memory | 16 GB | 736 GB/s | 21 (same) |
| RTX 5070 Ti 16GB | Same memory | 16 GB | 896 GB/s | 21 (same) |
| RTX 5080 16GB | Same memory | 16 GB | 960 GB/s | 21 (same) |
| RTX 5070 12GB | Next size down | 12 GB | 672 GB/s | 20 (−1) |
| 2× RTX 3060 12GB | Next size up | 24 GB | 720 GB/s | 38 (+17) |
Model numbers read from Hugging Face on .