Can I run Llama 3.1 8B on an RTX 4060 Ti 16GB?
Yes: Llama 3.1 8B needs about 9.88 GB at Q4_K_M with 32K tokens of context, which fits the RTX 4060 Ti 16GB with 6.12 GB to spare. At 32K the RTX 4060 Ti holds up to Q8_0 (13.6 GB), and Q4_K_M runs up to 72K tokens.
Yes Q4_K_M with 32K tokens of context
- Q4_K_M, 32K
- 9.88 GB
- RTX 4060 Ti
- 16 GB, 288 GB/s
- To spare
- 6.12 GB
- Tokens/s
- 17–23 tokens/s
At Q4_K_M with 32K tokens of context it writes about 17–23 tokens/s for one request on an RTX 4060 Ti 16GB.
Best precision for Llama 3.1 8B on an RTX 4060 Ti 16GB
The most precise setting that leaves at least 0.5 GB free; one that fits with less is marked tight.
| Context | Best fit | Memory | Free | Tokens/s |
|---|---|---|---|---|
| 8K | Q8_0 | 10.3 GB | 5.66 GB | 16–22 |
| 32K | Q8_0 | 13.6 GB | 2.36 GB | 12–17 |
| 128K (full) | Nothing fits | — | — | — |
Llama 3.1 8B on the RTX 4060 Ti as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 6.03 GB | 9.97 GB | 28–39 | 9.79 GB | 6.21 GB | 17–24 |
| 8K | 6.58 GB | 9.42 GB | 26–36 | 10.3 GB | 5.66 GB | 16–22 |
| 16K | 7.68 GB | 8.32 GB | 22–30 | 11.4 GB | 4.56 GB | 15–20 |
| 32K | 9.88 GB | 6.12 GB | 17–23 | 13.6 GB | 2.36 GB | 12–17 |
| 64K | 14.3 GB | 1.72 GB | 12–16 | 18.0 GB | −2.04 GB | — |
| 128K (full) | 23.1 GB | −7.08 GB | — | 26.8 GB | −10.8 GB | — |
Llama 3.1 8B on more than one RTX 4060 Ti
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s |
|---|---|---|---|---|
| 2× (32 GB) | BF16 | 15–21 | 128K (full) | 31–44 |
| 4× (64 GB) | BF16 | 28–40 | 128K (full) | 56–84 |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 8.92 GB at 32K, 7.08 GB under the RTX 4060 Ti; it fits with 0.5 GB to spare up to 79K tokens.
- Smallest setup for Q4_K_M at 32K: RTX 3060 12GB.
Run Llama 3.1 8B on the RTX 4060 Ti with llama-server
llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M -c 73728Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf, 4.9 GB, from bartowski/
Questions
Can I run Llama 3.1 8B on an RTX 4060 Ti 16GB?
Yes: Llama 3.1 8B needs about 9.88 GB at Q4_K_M with 32K tokens of context, which fits the RTX 4060 Ti 16GB with 6.12 GB to spare. At 32K the RTX 4060 Ti holds up to Q8_0 (13.6 GB), and Q4_K_M runs up to 72K tokens.
How fast is Llama 3.1 8B on an RTX 4060 Ti 16GB?
At Q4_K_M with 32K tokens of context it writes about 17–23 tokens/s for one request on an RTX 4060 Ti 16GB.
What does a second RTX 4060 Ti 16GB change for Llama 3.1 8B?
Two RTX 4060 Ti 16GB cards (32 GB in one tensor-parallel group) hold Llama 3.1 8B at BF16 with 32K, and Q4_K_M up to 128K (full) tokens, at about 31–44 tokens/s.
Try other settings in the VRAM calculator or the speed calculator. See also Llama 3.1 8B VRAM requirements, what LLMs an RTX 4060 Ti 16GB can run and every pair, or detect your own GPU. Model data checked .