Can I run Qwen3.8 Flash Next on an RTX 4060 Ti 16GB?
With --n-cpu-moe 42: the UD-Q4_K_XL GGUF keeps 15.5 GB on the RTX 4060 Ti and 63.1 GB in system RAM, with the experts of 42 of its 48 layers moved there. Whole, Qwen3.8 Flash Next needs about 113 GB at Q4_K_M with 32K tokens of context, 96.9 GB more than the RTX 4060 Ti 16GB holds. The smallest setup here that holds Qwen3.8 Flash Next at Q4_K_M with 32K is DGX Spark (128 GB, 120 GB usable).
With offload the UD-Q4_K_XL GGUF with --n-cpu-moe 42 and 32K tokens of context
- On the card
- 15.5 GB
- RTX 4060 Ti
- 16 GB, 288 GB/s
- In RAM
- 63.1 GB
- Tokens/s
- 8.5–14 tokens/s
With the UD-Q4_K_XL GGUF, --n-cpu-moe 42 and 32K tokens of context it writes about 8.5–14 tokens/s for one request on an RTX 4060 Ti 16GB and dual-channel DDR5-5600.
Qwen3.8 Flash Next on the RTX 4060 Ti as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 112 GB | −96.2 GB | — | 197 GB | −181 GB | — |
| 8K | 112 GB | −96.3 GB | — | 197 GB | −181 GB | — |
| 16K | 112 GB | −96.5 GB | — | 197 GB | −181 GB | — |
| 32K | 113 GB | −96.9 GB | — | 197 GB | −181 GB | — |
| 64K | 114 GB | −97.7 GB | — | 198 GB | −182 GB | — |
| 128K | 115 GB | −99.4 GB | — | 200 GB | −184 GB | — |
| 256K (full) | 119 GB | −103 GB | — | 203 GB | −187 GB | — |
--n-cpu-moe for Qwen3.8 Flash Next on the RTX 4060 Ti
With --n-cpu-moe 42 the UD-Q4_K_XL GGUF keeps 15.5 GB on the RTX 4060 Ti and 63.1 GB in RAM, about 8–14 tokens/s with DDR5-5600. Measured UD-Q4_K_XL file, 1 GB of buffers; tokens/s by system RAM speed.
Qwen3.8 Flash Next on more than one RTX 4060 Ti
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s |
|---|---|---|---|---|
| 2× (32 GB) | Nothing fits | — | Does not fit | — |
| 4× (64 GB) | Nothing fits | — | Does not fit | — |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 91.5 GB at 32K, 75.5 GB over the RTX 4060 Ti; it does not fit with 0.5 GB to spare even with 1K tokens.
- Smallest setup for Q4_K_M at 32K: DGX Spark (128 GB, 120 GB usable).
Run Qwen3.8 Flash Next on the RTX 4060 Ti with llama-server
llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL -c 32768 --n-cpu-moe 42UD-Q4_K_XL/
Questions
Can I run Qwen3.8 Flash Next on an RTX 4060 Ti 16GB?
With --n-cpu-moe 42: the UD-Q4_K_XL GGUF keeps 15.5 GB on the RTX 4060 Ti and 63.1 GB in system RAM, with the experts of 42 of its 48 layers moved there. Whole, Qwen3.8 Flash Next needs about 113 GB at Q4_K_M with 32K tokens of context, 96.9 GB more than the RTX 4060 Ti 16GB holds. The smallest setup here that holds Qwen3.8 Flash Next at Q4_K_M with 32K is DGX Spark (128 GB, 120 GB usable).
How fast is Qwen3.8 Flash Next on an RTX 4060 Ti 16GB?
With the UD-Q4_K_XL GGUF, --n-cpu-moe 42 and 32K tokens of context it writes about 8.5–14 tokens/s for one request on an RTX 4060 Ti 16GB and dual-channel DDR5-5600.
Does Qwen3.8 Flash Next need --n-cpu-moe on an RTX 4060 Ti 16GB?
With --n-cpu-moe 42 the UD-Q4_K_XL GGUF keeps 15.5 GB on the RTX 4060 Ti and 63.1 GB in RAM, about 8–14 tokens/s with DDR5-5600.
What does a second RTX 4060 Ti 16GB change for Qwen3.8 Flash Next?
Even two RTX 4060 Ti 16GB cards (32 GB) do not hold Qwen3.8 Flash Next at 32K.
Try other settings in the VRAM calculator, the speed calculator or the MoE offload planner. See also Qwen3.8 Flash Next VRAM requirements, what LLMs an RTX 4060 Ti 16GB can run and every pair, or detect your own GPU. Model data checked .