Can I run Llama 3.1 70B on an RTX 4090?
No: Llama 3.1 70B needs about 55.2 GB at Q4_K_M with 32K tokens of context, 31.2 GB more than the 24 GB RTX 4090 holds, so it takes 4 of them (96 GB) in one tensor-parallel group; three would hold it, but tensor parallel needs 2, 4 or 8 cards. The smallest setup here that holds Llama 3.1 70B at Q4_K_M with 32K is 2× RTX 5090 (64 GB).
No Q4_K_M with 32K tokens of context on 4 cards
- Q4_K_M, 32K
- 55.2 GB
- RTX 4090
- 24 GB, 1,008 GB/s
- Short by
- 31.2 GB
- Tokens/s
- 36–53 tokens/s
At Q4_K_M with 32K tokens of context on 4 cards it writes about 36–53 tokens/s for one request on 4 RTX 4090 cards.
Llama 3.1 70B on the RTX 4090 as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 45.6 GB | −21.6 GB | — | 78.7 GB | −54.7 GB | — |
| 8K | 47.0 GB | −23.0 GB | — | 80.0 GB | −56.0 GB | — |
| 16K | 49.7 GB | −25.7 GB | — | 82.8 GB | −58.8 GB | — |
| 32K | 55.2 GB | −31.2 GB | — | 88.3 GB | −64.3 GB | — |
| 64K | 66.2 GB | −42.2 GB | — | 99.3 GB | −75.3 GB | — |
| 128K (full) | 88.2 GB | −64.2 GB | — | 121 GB | −97.3 GB | — |
Llama 3.1 70B on more than one RTX 4090
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s | Rent per hour |
|---|---|---|---|---|---|
| 2× (48 GB) | Q3_K_M | 23–32 | 7K | — | $1.06 |
| 4× (96 GB) | Q8_0 | 24–34 | 128K (full) | 36–53 | $2.12 |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 46.8 GB at 32K, 22.8 GB over the RTX 4090; it does not fit with 0.5 GB to spare even with 1K tokens.
- Renting four RTX 4090 GPUs costs about $2.12 an hour (median on getdeploying.com, 2026-09-29): $11–16 per million tokens at 36–53 tokens/s.
- Smallest setup for Q4_K_M at 32K: 2× RTX 5090 (64 GB).
Run Llama 3.1 70B on the RTX 4090 with llama-server
llama-server -hf bartowski/Meta-Llama-3.1-70B-Instruct-GGUF:Q4_K_M -c 131072Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf, 42.5 GB, from bartowski/
Questions
Can I run Llama 3.1 70B on an RTX 4090?
No: Llama 3.1 70B needs about 55.2 GB at Q4_K_M with 32K tokens of context, 31.2 GB more than the 24 GB RTX 4090 holds, so it takes 4 of them (96 GB) in one tensor-parallel group; three would hold it, but tensor parallel needs 2, 4 or 8 cards. The smallest setup here that holds Llama 3.1 70B at Q4_K_M with 32K is 2× RTX 5090 (64 GB).
How fast is Llama 3.1 70B on an RTX 4090?
At Q4_K_M with 32K tokens of context on 4 cards it writes about 36–53 tokens/s for one request on 4 RTX 4090 cards.
What does a second RTX 4090 change for Llama 3.1 70B?
Two RTX 4090 cards (48 GB in one tensor-parallel group) hold Llama 3.1 70B at Q3_K_M with 32K, and Q4_K_M up to 7K tokens.
Try other settings in the VRAM calculator or the speed calculator. See also Llama 3.1 70B VRAM requirements, what LLMs an RTX 4090 can run and every pair, or detect your own GPU. Model data checked .