Can I run Qwen3.8 Flash Next on an RTX 4070 12GB?
With --n-cpu-moe 45: the UD-Q4_K_XL GGUF keeps 11.1 GB on the RTX 4070 and 67.5 GB in system RAM, with the experts of 45 of its 48 layers moved there. Whole, Qwen3.8 Flash Next needs about 113 GB at Q4_K_M with 32K tokens of context, 101 GB more than the RTX 4070 12GB holds. The smallest setup here that holds Qwen3.8 Flash Next at Q4_K_M with 32K is DGX Spark (128 GB, 120 GB usable).
With offload the UD-Q4_K_XL GGUF with --n-cpu-moe 45 and 32K tokens of context
- On the card
- 11.1 GB
- RTX 4070
- 12 GB, 504 GB/s
- In RAM
- 67.5 GB
- Tokens/s
- 11–18 tokens/s
With the UD-Q4_K_XL GGUF, --n-cpu-moe 45 and 32K tokens of context it writes about 11–18 tokens/s for one request on an RTX 4070 12GB and dual-channel DDR5-5600.
Qwen3.8 Flash Next on the RTX 4070 as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 112 GB | −100 GB | — | 197 GB | −185 GB | — |
| 8K | 112 GB | −100 GB | — | 197 GB | −185 GB | — |
| 16K | 112 GB | −100 GB | — | 197 GB | −185 GB | — |
| 32K | 113 GB | −101 GB | — | 197 GB | −185 GB | — |
| 64K | 114 GB | −102 GB | — | 198 GB | −186 GB | — |
| 128K | 115 GB | −103 GB | — | 200 GB | −188 GB | — |
| 256K (full) | 119 GB | −107 GB | — | 203 GB | −191 GB | — |
--n-cpu-moe for Qwen3.8 Flash Next on the RTX 4070
With --n-cpu-moe 45 the UD-Q4_K_XL GGUF keeps 11.1 GB on the RTX 4070 and 67.5 GB in RAM, about 11–18 tokens/s with DDR5-5600. Measured UD-Q4_K_XL file, 1 GB of buffers; tokens/s by system RAM speed.
Qwen3.8 Flash Next on more than one RTX 4070
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s |
|---|---|---|---|---|
| 2× (24 GB) | Nothing fits | — | Does not fit | — |
| 4× (48 GB) | Nothing fits | — | Does not fit | — |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 91.5 GB at 32K, 79.5 GB over the RTX 4070; it does not fit with 0.5 GB to spare even with 1K tokens.
- Smallest setup for Q4_K_M at 32K: DGX Spark (128 GB, 120 GB usable).
Run Qwen3.8 Flash Next on the RTX 4070 with llama-server
llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL -c 32768 --n-cpu-moe 45UD-Q4_K_XL/
Questions
Can I run Qwen3.8 Flash Next on an RTX 4070 12GB?
With --n-cpu-moe 45: the UD-Q4_K_XL GGUF keeps 11.1 GB on the RTX 4070 and 67.5 GB in system RAM, with the experts of 45 of its 48 layers moved there. Whole, Qwen3.8 Flash Next needs about 113 GB at Q4_K_M with 32K tokens of context, 101 GB more than the RTX 4070 12GB holds. The smallest setup here that holds Qwen3.8 Flash Next at Q4_K_M with 32K is DGX Spark (128 GB, 120 GB usable).
How fast is Qwen3.8 Flash Next on an RTX 4070 12GB?
With the UD-Q4_K_XL GGUF, --n-cpu-moe 45 and 32K tokens of context it writes about 11–18 tokens/s for one request on an RTX 4070 12GB and dual-channel DDR5-5600.
Does Qwen3.8 Flash Next need --n-cpu-moe on an RTX 4070 12GB?
With --n-cpu-moe 45 the UD-Q4_K_XL GGUF keeps 11.1 GB on the RTX 4070 and 67.5 GB in RAM, about 11–18 tokens/s with DDR5-5600.
What does a second RTX 4070 12GB change for Qwen3.8 Flash Next?
Even two RTX 4070 12GB cards (24 GB) do not hold Qwen3.8 Flash Next at 32K.
Try other settings in the VRAM calculator, the speed calculator or the MoE offload planner. See also Qwen3.8 Flash Next VRAM requirements, what LLMs an RTX 4070 12GB can run and every pair, or detect your own GPU. Model data checked .