Can I run Llama 3.1 8B on an RTX 4090?

Yes: Llama 3.1 8B needs about 9.88 GB at Q4_K_M with 32K tokens of context, which fits the 24 GB RTX 4090 with 14.1 GB to spare. At 32K the RTX 4090 holds up to BF16 (21.4 GB), and Q4_K_M runs up to 128K (full) tokens.

Yes Q4_K_M with 32K tokens of context

Q4_K_M, 32K: 9.88 GB
RTX 4090: 24 GB, 1,008 GB/s
To spare: 14.1 GB
Tokens/s: 56–79 tokens/s

At Q4_K_M with 32K tokens of context it writes about 56–79 tokens/s for one request on an RTX 4090.

Best precision for Llama 3.1 8B on an RTX 4090

The most precise setting that leaves at least 0.5 GB free; one that fits with less is marked tight.

Context	Best fit	Memory	Free	Tokens/s
8K	BF16	18.1 GB	5.95 GB	31–43
32K	BF16	21.4 GB	2.65 GB	26–36
128K (full)	Q4_K_M	23.1 GB	945 MB	24–34

Llama 3.1 8B on the RTX 4090 as the context fills

One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.

Context	Q4_K_M	Free	Tokens/s	Q8_0	Free	Tokens/s
4K	6.03 GB	18.0 GB	89–131	9.79 GB	14.2 GB	56–80
8K	6.58 GB	17.4 GB	82–120	10.3 GB	13.7 GB	53–76
16K	7.68 GB	16.3 GB	71–102	11.4 GB	12.6 GB	48–68
32K	9.88 GB	14.1 GB	56–79	13.6 GB	10.4 GB	41–57
64K	14.3 GB	9.72 GB	39–55	18.0 GB	5.96 GB	31–43
128K (full)	23.1 GB	945 MB	24–34	26.8 GB	−2.84 GB	—

Llama 3.1 8B on more than one RTX 4090

Cards	Best at 32K	Tokens/s	Q4_K_M longest	Q4_K_M tokens/s	Rent per hour
2× (48 GB)	BF16	46–68	128K (full)	85–136	$1.06
4× (96 GB)	BF16	79–125	128K (full)	131–231	$2.12

Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.

Other options

The next smaller setting, Q3_K_M, takes 8.92 GB at 32K, 15.1 GB under the RTX 4090; it fits with 0.5 GB to spare up to 128K (full) tokens.
Renting an RTX 4090 costs about $0.53 an hour (median on getdeploying.com, 2026-09-29): $1.9–2.7 per million tokens at 56–79 tokens/s.
Smallest setup for Q4_K_M at 32K: RTX 3060 12GB.

Run Llama 3.1 8B on the RTX 4090 with llama-server

llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M -c 131072

Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf, 4.9 GB, from bartowski/Meta-Llama-3.1-8B-Instruct-GGUF (checked 2026-09-29). At -c 131072 on the RTX 4090: 23.1 GB of 24 GB, 880 MB free.

Questions

Can I run Llama 3.1 8B on an RTX 4090?

How fast is Llama 3.1 8B on an RTX 4090?

At Q4_K_M with 32K tokens of context it writes about 56–79 tokens/s for one request on an RTX 4090.

What does a second RTX 4090 change for Llama 3.1 8B?

Two RTX 4090 cards (48 GB in one tensor-parallel group) hold Llama 3.1 8B at BF16 with 32K, and Q4_K_M up to 128K (full) tokens, at about 85–136 tokens/s.

Try other settings in the VRAM calculator or the speed calculator. See also Llama 3.1 8B VRAM requirements, what LLMs an RTX 4090 can run and every pair, or detect your own GPU. Model data checked 2026-09-26.