Can I run Qwen3.8 27B on an RTX 4070 12GB?
No: Qwen3.8 27B needs about 19.9 GB at Q4_K_M with 32K tokens of context, 7.92 GB more than the RTX 4070 12GB holds, so it takes 2 of them (24 GB) in one tensor-parallel group. The smallest setup here that holds Qwen3.8 27B at Q4_K_M with 32K is RTX 3090 (24 GB).
No Q4_K_M with 32K tokens of context on 2 cards
- Q4_K_M, 32K
- 19.9 GB
- RTX 4070
- 12 GB, 504 GB/s
- Short by
- 7.92 GB
- Tokens/s
- 27–38 tokens/s
At Q4_K_M with 32K tokens of context on 2 cards it writes about 27–38 tokens/s for one request on 2 RTX 4070 12GB cards.
Qwen3.8 27B on the RTX 4070 as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 18.0 GB | −5.99 GB | — | 31.0 GB | −19.0 GB | — |
| 8K | 18.3 GB | −6.27 GB | — | 31.3 GB | −19.3 GB | — |
| 16K | 18.8 GB | −6.82 GB | — | 31.8 GB | −19.8 GB | — |
| 32K | 19.9 GB | −7.92 GB | — | 32.9 GB | −20.9 GB | — |
| 64K | 22.1 GB | −10.1 GB | — | 35.1 GB | −23.1 GB | — |
| 128K | 26.5 GB | −14.5 GB | — | 39.5 GB | −27.5 GB | — |
| 256K (full) | 35.3 GB | −23.3 GB | — | 48.3 GB | −36.3 GB | — |
Qwen3.8 27B on more than one RTX 4070
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s |
|---|---|---|---|---|
| 2× (24 GB) | Q5_K_M | 23–33 | 76K | 27–38 |
| 4× (48 GB) | Q8_0 | 31–45 | 256K (full) | 49–72 |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 16.6 GB at 32K, 4.61 GB over the RTX 4070; it does not fit with 0.5 GB to spare even with 1K tokens.
- Smallest setup for Q4_K_M at 32K: RTX 3090 (24 GB).
Run Qwen3.8 27B on the RTX 4070 with llama-server
llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M -c 45056Qwen3.8-27B-Q4_K_M.gguf, 19.0 GB, from ggml-org/
Questions
Can I run Qwen3.8 27B on an RTX 4070 12GB?
No: Qwen3.8 27B needs about 19.9 GB at Q4_K_M with 32K tokens of context, 7.92 GB more than the RTX 4070 12GB holds, so it takes 2 of them (24 GB) in one tensor-parallel group. The smallest setup here that holds Qwen3.8 27B at Q4_K_M with 32K is RTX 3090 (24 GB).
How fast is Qwen3.8 27B on an RTX 4070 12GB?
At Q4_K_M with 32K tokens of context on 2 cards it writes about 27–38 tokens/s for one request on 2 RTX 4070 12GB cards.
What does a second RTX 4070 12GB change for Qwen3.8 27B?
Two RTX 4070 12GB cards (24 GB in one tensor-parallel group) hold Qwen3.8 27B at Q5_K_M with 32K, and Q4_K_M up to 76K tokens, at about 27–38 tokens/s.
Try other settings in the VRAM calculator or the speed calculator. See also Qwen3.8 27B VRAM requirements, what LLMs an RTX 4070 12GB can run and every pair, or detect your own GPU. Model data checked .