Can I run Gemma 4 31B on an RTX 3060 12GB?
No: Gemma 4 31B needs about 23.9 GB at Q4_K_M with 32K tokens of context, 11.9 GB more than the RTX 3060 12GB holds, so it takes 4 of them (48 GB) in one tensor-parallel group; three would hold it, but tensor parallel needs 2, 4 or 8 cards. The smallest setup here that holds Gemma 4 31B at Q4_K_M with 32K is RTX 3090 (24 GB).
No Q4_K_M with 32K tokens of context on 4 cards
- Q4_K_M, 32K
- 23.9 GB
- RTX 3060
- 12 GB, 360 GB/s
- Short by
- 11.9 GB
- Tokens/s
- 31–45 tokens/s
At Q4_K_M with 32K tokens of context on 4 cards it writes about 31–45 tokens/s for one request on 4 RTX 3060 12GB cards.
Gemma 4 31B on the RTX 3060 as the context fills
One request, FP16 KV cache, 0.5 GB plus 10% overhead; a minus sign is memory missing, and tight is less than 0.5 GB free.
| Context | Q4_K_M | Free | Tokens/s | Q8_0 | Free | Tokens/s |
|---|---|---|---|---|---|---|
| 4K | 21.5 GB | −9.52 GB | — | 36.2 GB | −24.2 GB | — |
| 8K | 21.9 GB | −9.86 GB | — | 36.5 GB | −24.5 GB | — |
| 16K | 22.5 GB | −10.5 GB | — | 37.2 GB | −25.2 GB | — |
| 32K | 23.9 GB | −11.9 GB | — | 38.6 GB | −26.6 GB | — |
| 64K | 26.7 GB | −14.7 GB | — | 41.3 GB | −29.3 GB | — |
| 128K | 32.2 GB | −20.2 GB | — | 46.8 GB | −34.8 GB | — |
| 256K (full) | 43.2 GB | −31.2 GB | — | 57.8 GB | −45.8 GB | — |
Gemma 4 31B on more than one RTX 3060
| Cards | Best at 32K | Tokens/s | Q4_K_M longest | Q4_K_M tokens/s | Rent per hour |
|---|---|---|---|---|---|
| 2× (24 GB) | Q3_K_M | 19–27 | 21K | — | $0.16 |
| 4× (48 GB) | Q8_0 | 20–28 | 256K (full) | 31–45 | $0.32 |
Tensor parallel with the combined bandwidth, as in vLLM; llama.cpp splits layers by default and runs at about one card's speed.
Other options
- The next smaller setting, Q3_K_M, takes 20.2 GB at 32K, 8.20 GB over the RTX 3060; it does not fit with 0.5 GB to spare even with 1K tokens.
- Renting four RTX 3060 12GB GPUs costs about $0.32 an hour (median on getdeploying.com, 2026-09-29): $2.0–2.9 per million tokens at 31–45 tokens/s.
- Smallest setup for Q4_K_M at 32K: RTX 3090 (24 GB).
Run Gemma 4 31B on the RTX 3060 with llama-server
llama-server -hf unsloth/gemma-4-31B-it-GGUF:Q4_K_M -c 262144 -np 1gemma-4-31B-it-Q4_K_M.gguf, 18.3 GB, from unsloth/
Questions
Can I run Gemma 4 31B on an RTX 3060 12GB?
No: Gemma 4 31B needs about 23.9 GB at Q4_K_M with 32K tokens of context, 11.9 GB more than the RTX 3060 12GB holds, so it takes 4 of them (48 GB) in one tensor-parallel group; three would hold it, but tensor parallel needs 2, 4 or 8 cards. The smallest setup here that holds Gemma 4 31B at Q4_K_M with 32K is RTX 3090 (24 GB).
How fast is Gemma 4 31B on an RTX 3060 12GB?
At Q4_K_M with 32K tokens of context on 4 cards it writes about 31–45 tokens/s for one request on 4 RTX 3060 12GB cards.
What does a second RTX 3060 12GB change for Gemma 4 31B?
Two RTX 3060 12GB cards (24 GB in one tensor-parallel group) hold Gemma 4 31B at Q3_K_M with 32K, and Q4_K_M up to 21K tokens.
Try other settings in the VRAM calculator or the speed calculator. See also Gemma 4 31B VRAM requirements, what LLMs an RTX 3060 12GB can run and every pair, or detect your own GPU. Model data checked .