Can I run Llama 3.1 8B on an RTX 3090?
Yes: Llama 3.1 8B needs about 9.88 GB at Q4_K_M with 32K tokens of context, which fits the 24 GB RTX 3090 with 14.1 GB to spare. At 32K the RTX 3090 holds up to BF16 (21.4 GB), and Q4_K_M runs up to 128K (full) tokens.
Yes Q4_K_M with 32K tokens of context
- Q4_K_M, 32K
- 9.88 GB
- RTX 3090
- 24 GB, 936 GB/s
- To spare
- 14.1 GB
- Tokens/s
- 52–74 tokens/s
At Q4_K_M with 32K tokens of context it writes about 52–74 tokens/s for one request on an RTX 3090.
Best precision for Llama 3.1 8B on an RTX 3090
The most precise setting that leaves at least 0.5 GB free; one that fits with less is marked tight.
| Context | Best fit | Memory | Free | Tokens/s |
|---|---|---|---|---|
| 8K | BF16 | 18.1 GB | 5.95 GB | 29–40 |
| 32K | BF16 | 21.4 GB | 2.65 GB | 24–34 |
| 128K (full) | Q4_K_M | 23.1 GB | 945 MB | 23–31 |
Llama 3.1 8B on the RTX 3090 as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 6.03 GB | 18.0 GB | 83–122 | 9.79 GB | 14.2 GB | 52–75 |
| 8K | 6.58 GB | 17.4 GB | 77–112 | 10.3 GB | 13.7 GB | 50–71 |
| 16K | 7.68 GB | 16.3 GB | 66–95 | 11.4 GB | 12.6 GB | 45–64 |
| 32K | 9.88 GB | 14.1 GB | 52–74 | 13.6 GB | 10.4 GB | 38–53 |
| 64K | 14.3 GB | 9.72 GB | 36–51 | 18.0 GB | 5.96 GB | 29–40 |
| 128K (full) | 23.1 GB | 945 MB | 23–31 | 26.8 GB | −2.84 GB | — |
Llama 3.1 8B on more than one RTX 3090
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s | Rent per hour |
|---|---|---|---|---|---|
| 2× (48 GB) | BF16 | 43–63 | 128K (full) | 81–128 | $0.54 |
| 4× (96 GB) | BF16 | 75–117 | 128K (full) | 126–219 | $1.08 |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 8.92 GB at 32K, 15.1 GB under the RTX 3090; it fits with 0.5 GB to spare up to 128K (full) tokens.
- Renting an RTX 3090 costs about $0.27 an hour (median on getdeploying.com, 2026-09-29): $1.0–1.4 per million tokens at 52–74 tokens/s.
- Smallest setup for Q4_K_M at 32K: RTX 3060 12GB.
Run Llama 3.1 8B on the RTX 3090 with llama-server
llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M -c 131072 -ngl 99Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf, 4.9 GB, from bartowski/
Questions
Can I run Llama 3.1 8B on an RTX 3090?
Yes: Llama 3.1 8B needs about 9.88 GB at Q4_K_M with 32K tokens of context, which fits the 24 GB RTX 3090 with 14.1 GB to spare. At 32K the RTX 3090 holds up to BF16 (21.4 GB), and Q4_K_M runs up to 128K (full) tokens.
How fast is Llama 3.1 8B on an RTX 3090?
At Q4_K_M with 32K tokens of context it writes about 52–74 tokens/s for one request on an RTX 3090.
What does a second RTX 3090 change for Llama 3.1 8B?
Two RTX 3090 cards (48 GB in one tensor-parallel group) hold Llama 3.1 8B at BF16 with 32K, and Q4_K_M up to 128K (full) tokens, at about 81–128 tokens/s.
Try other settings in the VRAM calculator or the speed calculator. See also Llama 3.1 8B VRAM requirements, what LLMs an RTX 3090 can run and every pair, or detect your own GPU. Model data checked .