ToolSite

What LLMs can an RTX 3060 12GB run?

An RTX 3060 12GB gives a model 12 GB of memory and 360 GB/s of bandwidth. Of the 40 open models tracked here, 7 fit at Q4_K_M with an 8,192-token context, and 2 more at a lower precision.

Models that fit on one RTX 3060 12GB

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first.

Model Best precision Memory Longest context Tokens/s
Gemma 4 26B-A4B 25.8B, 4B active GGUF Q2_K 11.9 GB 19K 51–87
gpt-oss-20b 20.9B, 3.6B active GGUF Q3_K_M 11.2 GB 39K 51–88
Gemma 4 12B 12.0B GGUF Q6_K 11.0 GB 128K 19–26
Qwen3.5 9B 9.7B GGUF Q8_0 11.3 GB 28K 18–25
Qwen3 8B 8.2B GGUF Q8_0 10.7 GB 16K 19–27
Llama 3.1 8B 8.0B GGUF Q8_0 10.3 GB 19K 20–28
Gemma 4 E4B 8.0B GGUF Q8_0 9.36 GB 128K (full) 22–31
Nemotron 3 Nano 4B 4.0B FP16 / BF16 8.78 GB 195K 24–33
MiniCPM5 2B 2.5B FP16 / BF16 6.02 GB 128K (full) 35–49

Too big for one card

How many RTX 3060 12GB cards these need at Q4_K_M in one tensor-parallel group; some need more than 8.

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Other GPUs and Macs

Model numbers read from Hugging Face on .