Llama 3.1 70B VRAM requirements

Can I run Llama 3.1 70B on…

• RTX 4090: No, 4 cards
• RTX 3090: No, 4 cards
• RTX 5090: No, 2 cards
• RTX 3060 12GB: No, 8 cards
• RTX 4060 Ti 16GB: No, 4 cards
• RTX 5060 Ti 16GB: No, 4 cards
• RTX 4070 12GB: No, 8 cards
• RTX 5080 16GB: No, 4 cards
• RTX 4080 Super 16GB: No, 4 cards
• RX 7900 XTX: No, 4 cards
• M4 Max Mac (128 GB): Yes, up to Q8_0
• H100 SXM: Yes, up to Q6_K

How much VRAM does Llama 3.1 70B need?

Llama 3.1 70B needs 47.0 GB at Q4_K_M, 80.0 GB at Q8_0 and 148 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.

Fine-tuning Llama 3.1 70B? Llama 3.1 70B VRAM for LoRA, QLoRA and full training.

Llama 3.1 70B GGUF files on Hugging Face

The 7 GGUF files llama.cpp picks for Llama 3.1 70B with -hf <repo>:<QUANT>, with the bytes the Hugging Face file list reported on 2026-09-29. Each total is that file plus the FP16 KV cache and 0.5 GB + 10%, one request. "vs estimate" compares the file with the bits-per-weight size the table above uses.

GB here is GiB (1024³ bytes), as everywhere on this site; Hugging Face shows the Q4_K_M file as 42.5 GB, in 10⁹ bytes.

Which Llama 3.1 70B GGUF fits a 16, 24, 32, 48 or 80 GB GPU or a Mac

From the file sizes above: the largest file that fits each machine with an 8K context and 0.5 GB left free, the longest context it then has room for, and the longest context for Q4_K_M (39.6 GB). One request, FP16 KV cache. The Q4_K_M file is 157 MB under its bits-per-weight estimate; the estimate tables and the llama-server command on this page count the estimate, so they show less context.

Run Llama 3.1 70B with llama.cpp or Ollama

llama-server -hf bartowski/Meta-Llama-3.1-70B-Instruct-GGUF:Q4_K_M -c 9216

Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf, 42.5 GB, from bartowski/Meta-Llama-3.1-70B-Instruct-GGUF (checked 2026-09-29). -c 9216 is the longest Q4_K_M context on the M4 Pro Mac (64 GB, 48 GB usable).

ollama run llama3.1:70b

Ollama library tag llama3.1:70b; its quant is Ollama's choice. Same context: /set parameter num_ctx 9216.

Llama 3.1 70B at 8K, 32K and 128K (full) tokens of context

All 80 layers use full attention. Each extra token of context adds 320 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.

How fast Llama 3.1 70B writes

Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth. A dash means it does not fit on one card. Try other settings in the speed calculator.

Longest context on one GPU

How many tokens of context Llama 3.1 70B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.

Model details

Parameters

70.6B (70,553,706,496)

Layers

All 80 layers use full attention

Attention cache

8 KV heads × 128

Context length

131,072 tokens

Published weights

131 GB (BF16)

On Hugging Face

meta-llama/Llama-3.1-70B-Instruct

Reported benchmarks

GPQA, 0-shot 46.7 · MMLU-Pro (CoT), 5-shot 66.4. Vendor-reported: from the maker’s own model card, not rerun here; read 2026-09-30.

Llama 3.1 70B next to similar-sized models

The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.