Can I run Gemma 4 12B on an RTX 4090?

Yes: Gemma 4 12B needs about 8.98 GB at Q4_K_M with 32K tokens of context, which fits the 24 GB RTX 4090 with 15.0 GB to spare. At 32K the RTX 4090 holds up to Q8_0 (14.6 GB), and Q4_K_M runs up to 256K (full) tokens.

Yes Q4_K_M with 32K tokens of context

Q4_K_M, 32K: 8.98 GB
RTX 4090: 24 GB, 1,008 GB/s
To spare: 15.0 GB
Tokens/s: 61–87 tokens/s

At Q4_K_M with 32K tokens of context it writes about 61–87 tokens/s for one request on an RTX 4090.

Best precision for Gemma 4 12B on an RTX 4090

The most precise setting that leaves at least 0.5 GB free; one that fits with less is marked tight.

Context	Best fit	Memory	Free	Tokens/s
8K	Q8_0	14.2 GB	9.83 GB	39–55
32K	Q8_0	14.6 GB	9.42 GB	38–54
128K	Q8_0	16.2 GB	7.77 GB	34–48
256K (full)	Q8_0	18.4 GB	5.57 GB	30–42

Gemma 4 12B on the RTX 4090 as the context fills

One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.

Context	Q4_K_M	Free	Tokens/s	Q8_0	Free	Tokens/s
4K	8.50 GB	15.5 GB	64–92	14.1 GB	9.90 GB	39–55
8K	8.57 GB	15.4 GB	64–92	14.2 GB	9.83 GB	39–55
16K	8.70 GB	15.3 GB	63–90	14.3 GB	9.69 GB	39–55
32K	8.98 GB	15.0 GB	61–87	14.6 GB	9.42 GB	38–54
64K	9.53 GB	14.5 GB	57–82	15.1 GB	8.87 GB	37–52
128K	10.6 GB	13.4 GB	52–74	16.2 GB	7.77 GB	34–48
256K (full)	12.8 GB	11.2 GB	43–61	18.4 GB	5.57 GB	30–42

Gemma 4 12B on more than one RTX 4090

Cards	Best at 32K	Tokens/s	Q4_K_M longest	Q4_K_M tokens/s	Rent per hour
2× (48 GB)	BF16	38–56	256K (full)	91–148	$1.06
4× (96 GB)	BF16	68–105	256K (full)	138–248	$2.12

Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.

Other options

The next smaller setting, Q3_K_M, takes 7.55 GB at 32K, 16.4 GB under the RTX 4090; it fits with 0.5 GB to spare up to 256K (full) tokens.
Renting an RTX 4090 costs about $0.53 an hour (median on getdeploying.com, 2026-09-29): $1.7–2.4 per million tokens at 61–87 tokens/s.
Smallest setup for Q4_K_M at 32K: RTX 3060 12GB.

Run Gemma 4 12B on the RTX 4090 with llama-server

llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_M -c 262144 -ngl 99 -np 1

gemma-4-12b-it-Q4_K_M.gguf, 7.1 GB, from unsloth/gemma-4-12b-it-GGUF (checked 2026-09-29). At -c 262144 on the RTX 4090: 12.8 GB of 24 GB, 11.2 GB free. -np 1: one slot, one sliding window.

Questions

Can I run Gemma 4 12B on an RTX 4090?

How fast is Gemma 4 12B on an RTX 4090?

At Q4_K_M with 32K tokens of context it writes about 61–87 tokens/s for one request on an RTX 4090.

What does a second RTX 4090 change for Gemma 4 12B?

Two RTX 4090 cards (48 GB in one tensor-parallel group) hold Gemma 4 12B at BF16 with 32K, and Q4_K_M up to 256K (full) tokens, at about 91–148 tokens/s.

Try other settings in the VRAM calculator or the speed calculator. See also Gemma 4 12B VRAM requirements, what LLMs an RTX 4090 can run and every pair, or detect your own GPU. Model data checked 2026-09-26.