Can I run Qwen3.8 27B on an RTX 5060 Ti 16GB?
Only at Q2_K: Qwen3.8 27B needs about 19.9 GB at Q4_K_M with 32K tokens of context, 3.92 GB more than the RTX 5060 Ti 16GB holds, but 14.6 GB at Q2_K, which fits with 1.38 GB to spare. The smallest setup here that holds Qwen3.8 27B at Q4_K_M with 32K is RTX 3090 (24 GB).
Partly Q2_K with 32K tokens of context
- Q4_K_M, 32K
- 19.9 GB
- RTX 5060 Ti
- 16 GB, 448 GB/s
- Short by
- 3.92 GB
- Tokens/s
- 17–24 tokens/s
At Q2_K with 32K tokens of context it writes about 17–24 tokens/s for one request on an RTX 5060 Ti 16GB.
Best precision for Qwen3.8 27B on an RTX 5060 Ti 16GB
The most precise setting that leaves at least 0.5 GB free; one that fits with less is marked tight.
| Context | Best fit | Memory | Free | Tokens/s |
|---|---|---|---|---|
| 8K | Q3_K_M | 15.0 GB | 1.04 GB | 17–24 |
| 32K | Q2_K | 14.6 GB | 1.38 GB | 17–24 |
| 128K | Nothing fits | — | — | — |
| 256K (full) | Nothing fits | — | — | — |
Qwen3.8 27B on the RTX 5060 Ti as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 18.0 GB | −1.99 GB | — | 31.0 GB | −15.0 GB | — |
| 8K | 18.3 GB | −2.27 GB | — | 31.3 GB | −15.3 GB | — |
| 16K | 18.8 GB | −2.82 GB | — | 31.8 GB | −15.8 GB | — |
| 32K | 19.9 GB | −3.92 GB | — | 32.9 GB | −16.9 GB | — |
| 64K | 22.1 GB | −6.12 GB | — | 35.1 GB | −19.1 GB | — |
| 128K | 26.5 GB | −10.5 GB | — | 39.5 GB | −23.5 GB | — |
| 256K (full) | 35.3 GB | −19.3 GB | — | 48.3 GB | −32.3 GB | — |
Qwen3.8 27B on more than one RTX 5060 Ti
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s |
|---|---|---|---|---|
| 2× (32 GB) | Q6_K | 18–26 | 192K | 24–34 |
| 4× (64 GB) | BF16 | 16–23 | 256K (full) | 44–65 |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, IQ3_XXS, takes 14.4 GB at 32K, 1.56 GB under the RTX 5060 Ti; it fits with 0.5 GB to spare up to 47K tokens.
- Smallest setup for Q4_K_M at 32K: RTX 3090 (24 GB).
Run Qwen3.8 27B on the RTX 5060 Ti with llama-server
llama-server -hf bartowski/Qwen3.8-27B-GGUF:Q2_K -c 45056Qwen3.8-27B-Q2_K.gguf, 10.8 GB, from bartowski/
Questions
Can I run Qwen3.8 27B on an RTX 5060 Ti 16GB?
Only at Q2_K: Qwen3.8 27B needs about 19.9 GB at Q4_K_M with 32K tokens of context, 3.92 GB more than the RTX 5060 Ti 16GB holds, but 14.6 GB at Q2_K, which fits with 1.38 GB to spare. The smallest setup here that holds Qwen3.8 27B at Q4_K_M with 32K is RTX 3090 (24 GB).
How fast is Qwen3.8 27B on an RTX 5060 Ti 16GB?
At Q2_K with 32K tokens of context it writes about 17–24 tokens/s for one request on an RTX 5060 Ti 16GB.
What does a second RTX 5060 Ti 16GB change for Qwen3.8 27B?
Two RTX 5060 Ti 16GB cards (32 GB in one tensor-parallel group) hold Qwen3.8 27B at Q6_K with 32K, and Q4_K_M up to 192K tokens, at about 24–34 tokens/s.
Try other settings in the VRAM calculator or the speed calculator. See also Qwen3.8 27B VRAM requirements, what LLMs an RTX 5060 Ti 16GB can run and every pair, or detect your own GPU. Model data checked .