Can I run Llama 3.1 70B on an RX 7900 XTX?
No: Llama 3.1 70B needs about 55.2 GB at Q4_K_M with 32K tokens of context, 31.2 GB more than the 24 GB RX 7900 XTX holds, so it takes 4 of them (96 GB) in one tensor-parallel group; three would hold it, but tensor parallel needs 2, 4 or 8 cards. The smallest setup here that holds Llama 3.1 70B at Q4_K_M with 32K is 2× RTX 5090 (64 GB).
No Q4_K_M with 32K tokens of context on 4 cards
- Q4_K_M, 32K
- 55.2 GB
- RX 7900 XTX
- 24 GB, 960 GB/s
- Short by
- 31.2 GB
- Tokens/s
- 35–50 tokens/s
At Q4_K_M with 32K tokens of context on 4 cards it writes about 35–50 tokens/s for one request on 4 RX 7900 XTX cards.
Llama 3.1 70B on the RX 7900 XTX as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 45.6 GB | −21.6 GB | — | 78.7 GB | −54.7 GB | — |
| 8K | 47.0 GB | −23.0 GB | — | 80.0 GB | −56.0 GB | — |
| 16K | 49.7 GB | −25.7 GB | — | 82.8 GB | −58.8 GB | — |
| 32K | 55.2 GB | −31.2 GB | — | 88.3 GB | −64.3 GB | — |
| 64K | 66.2 GB | −42.2 GB | — | 99.3 GB | −75.3 GB | — |
| 128K (full) | 88.2 GB | −64.2 GB | — | 121 GB | −97.3 GB | — |
Llama 3.1 70B on more than one RX 7900 XTX
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s |
|---|---|---|---|---|
| 2× (48 GB) | Q3_K_M | 22–31 | 7K | — |
| 4× (96 GB) | Q8_0 | 23–32 | 128K (full) | 35–50 |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 46.8 GB at 32K, 22.8 GB over the RX 7900 XTX; it does not fit with 0.5 GB to spare even with 1K tokens.
- Smallest setup for Q4_K_M at 32K: 2× RTX 5090 (64 GB).
Run Llama 3.1 70B on the RX 7900 XTX with llama-server
llama-server -hf bartowski/Meta-Llama-3.1-70B-Instruct-GGUF:Q4_K_M -c 131072Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf, 42.5 GB, from bartowski/
Questions
Can I run Llama 3.1 70B on an RX 7900 XTX?
No: Llama 3.1 70B needs about 55.2 GB at Q4_K_M with 32K tokens of context, 31.2 GB more than the 24 GB RX 7900 XTX holds, so it takes 4 of them (96 GB) in one tensor-parallel group; three would hold it, but tensor parallel needs 2, 4 or 8 cards. The smallest setup here that holds Llama 3.1 70B at Q4_K_M with 32K is 2× RTX 5090 (64 GB).
How fast is Llama 3.1 70B on an RX 7900 XTX?
At Q4_K_M with 32K tokens of context on 4 cards it writes about 35–50 tokens/s for one request on 4 RX 7900 XTX cards.
What does a second RX 7900 XTX change for Llama 3.1 70B?
Two RX 7900 XTX cards (48 GB in one tensor-parallel group) hold Llama 3.1 70B at Q3_K_M with 32K, and Q4_K_M up to 7K tokens.
Try other settings in the VRAM calculator or the speed calculator. See also Llama 3.1 70B VRAM requirements, what LLMs an RX 7900 XTX can run and every pair, or detect your own GPU. Model data checked .