Nemotron 3 Super 120B-A12B VRAM requirements
Nemotron 3 Super 120B-A12B needs about 77.2 GB of VRAM at Q4_K_M with 8K context, so it needs an 80 GB card such as the A100 80GB or the H100 SXM; a 24 GB card is too small. Nemotron 3 Super 120B-A12B has 123.6B parameters, of which about 12B are used per token; all 512 experts still have to be in memory. With the same 8K context and one request it needs 127 GB at FP8 and 254 GB at FP16/BF16. The published weights take 230 GB (BF16). It does not fit on a 24 GB card at Q4_K_M; the smallest setup here that holds it with an 8K context is the A100 80GB.
77.2 GB at Q4_K_M, 8K context, one request
- FP8
- 127 GB
- FP16 / BF16
- 254 GB
- Published weights
- 230 GB
- Smallest setup, Q4_K_M
- A100 80GB
Worked example: Nemotron 3 Super 120B-A12B with 32K tokens
Inputs: Nemotron 3 Super 120B-A12B · Q4_K_M weights · 32,768 tokens of context · one request · FP16 KV cache
- Weights (123.6B at Q4_K_M)
- 69.6 GB
- KV cache (8 KB per token)
- 256 MB
- Buffers and runtime (0.5 GB + 10%)
- 7.49 GB
- Total
- 77.4 GB
The smallest setup here that holds it is the A100 80GB, with about 2.61 GB to spare. Change the inputs in the calculator
What makes Nemotron 3 Super 120B-A12B's memory use different
80 of its 88 layers keep a fixed-size Mamba or feed-forward state, so the cache grows by 8 KB per token where caching every layer would take 88 KB.
Per token of context it adds 8 KB of FP16 cache; Mistral Medium 3.5 128B (127.7B), the nearest-sized model here with plain full attention, adds 352 KB, so Nemotron 3 Super 120B-A12B needs 2% as much.
As a mixture-of-experts model it reads about 6.76 GB of its 69.6 GB Q4_K_M weights per generated token (10%), so it writes like a much smaller model while needing memory for all of them.
How much VRAM does Nemotron 3 Super 120B-A12B need?
Nemotron 3 Super 120B-A12B needs 77.2 GB at Q4_K_M, 135 GB at Q8_0 and 254 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.
| Precision | Weights | Total | Smallest setup |
|---|---|---|---|
| As published (BF16) | 230 GB | 254 GB | M3 Ultra Mac Studio (512 GB, 384 GB usable) |
| FP16 / BF16 | 230 GB | 254 GB | M3 Ultra Mac Studio (512 GB, 384 GB usable) |
| FP8 / INT8 | 115 GB | 127 GB | H200 (141 GB) |
| INT4 (AWQ / GPTQ) | 61.2 GB | 67.8 GB | A100 80GB |
| GGUF Q8_0 | 122 GB | 135 GB | H200 (141 GB) |
| GGUF Q6_K | 94.4 GB | 104 GB | DGX Spark (128 GB, 120 GB usable) |
| GGUF Q5_K_M | 81.6 GB | 90.3 GB | Ryzen AI Max+ 395 (128 GB, 96 GB usable) |
| GGUF Q4_K_M | 69.6 GB | 77.2 GB | A100 80GB |
| GGUF IQ4_XS | 62.6 GB | 69.4 GB | A100 80GB |
| GGUF Q3_K_M | 56.3 GB | 62.5 GB | 2× RTX 5090 (64 GB) |
| GGUF IQ3_XXS | 47.5 GB | 52.8 GB | 2× RTX 5090 (64 GB) |
| GGUF Q2_K | 48.2 GB | 53.6 GB | 2× RTX 5090 (64 GB) |
Fine-tuning Nemotron 3 Super 120B-A12B? Nemotron 3 Super 120B-A12B VRAM for LoRA, QLoRA and full training.
Nemotron 3 Super 120B-A12B GGUF files on Hugging Face
The 8 GGUF files llama.cpp picks for Nemotron 3 Super 120B-A12B with -hf <repo>:<QUANT>, with the bytes the Hugging Face file list reported on 2026-09-29. Each total is that file plus the FP16 KV cache and 0.5 GB + 10%, one request. "vs estimate" compares the file with the bits-per-weight size the table above uses.
| Quant | File | Size | vs estimate | 8K | 32K | 128K |
|---|---|---|---|---|---|---|
| Q2_K | bartowski/ | 51.1 GB54,819,768,960 bytes | +6% | 56.7 GB | 56.9 GB | 57.8 GB |
| Q3_K_M | unsloth/ | 57.5 GB61,712,254,816 bytes | +2% | 63.8 GB | 64.0 GB | 64.8 GB |
| IQ4_XS | unsloth/ | 60.1 GB64,484,689,760 bytes | −4% | 66.6 GB | 66.8 GB | 67.7 GB |
| Q4_K_M | unsloth/ | 76.9 GB82,541,168,480 bytes | +10% | 85.1 GB | 85.3 GB | 86.2 GB |
| Q5_K_M | unsloth/ | 100.0 GB107,333,699,552 bytes | +23% | 111 GB | 111 GB | 112 GB |
| Q6_K | unsloth/ | 107 GB114,751,457,216 bytes | +13% | 118 GB | 118 GB | 119 GB |
| Q8_0 | unsloth/ | 120 GB128,472,991,680 bytes | −2% | 132 GB | 132 GB | 133 GB |
| BF16 | unsloth/ | 225 GB241,518,659,424 bytes | −2% | 248 GB | 248 GB | 249 GB |
GB here is GiB (1024³ bytes), as everywhere on this site; Hugging Face shows the Q4_K_M file as 82.5 GB, in 10⁹ bytes.
Which Nemotron 3 Super 120B-A12B GGUF fits a 16, 24, 32, 48 or 80 GB GPU or a Mac
From the file sizes above: the largest file that fits each machine with an 8K context and 0.5 GB left free, the longest context it then has room for, and the longest context for Q4_K_M (76.9 GB). One request, FP16 KV cache.
| Hardware | Largest GGUF, 8K | Its longest context | Q4_K_M, longest context |
|---|---|---|---|
| RTX 5060 Ti 16GB | None fits | — | No |
| RTX 4090 (24 GB) | None fits | — | No |
| RTX 5090 (32 GB) | None fits | — | No |
| L40S (48 GB) | None fits | — | No |
| H100 SXM (80 GB) | IQ4_XS, 60.1 GB | 256K (full) | No |
| M4 Pro Mac (64 GB, 48 GB usable) | None fits | — | No |
| M4 Max Mac (128 GB, 96 GB usable) | Q4_K_M, 76.9 GB | 256K (full) | 256K (full) |
Nemotron 3 Super 120B-A12B at 8K, 32K, 128K and 256K (full) tokens of context
8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache. Each extra token of context adds 8 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.
How fast Nemotron 3 Super 120B-A12B writes
Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth and the parameters read per token. A dash means it does not fit on one card. Try other settings in the speed calculator.
| Hardware | Bandwidth | Q4_K_M | FP8 |
|---|---|---|---|
| RTX 3060 12GB | 360 GB/s | — | — |
| RTX 4090 | 1,008 GB/s | — | — |
| RTX 5090 | 1,792 GB/s | — | — |
| M4 Max Mac (128 GB) | 546 GB/s | 22–37 | — |
| M3 Ultra Mac Studio (512 GB) | 819 GB/s | 32–54 | 20–33 |
| H100 SXM | 3,350 GB/s | 114–205 | — |
| H200 | 4,800 GB/s | 152–281 | 101–181 |
Longest context on one GPU
How many tokens of context Nemotron 3 Super 120B-A12B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.
| GPU | Memory | Q4_K_M | Q8_0 | FP8 |
|---|---|---|---|---|
| RTX 3060 | 12 GB | No | No | No |
| RTX 4060 Ti 16GB | 16 GB | No | No | No |
| RTX 3090 / 4090 | 24 GB | No | No | No |
| RTX 5090 | 32 GB | No | No | No |
| A100 40GB | 40 GB | No | No | No |
| Mac, 64 GB unified memory | 48 GB | No | No | No |
| L40S / RTX 6000 Ada | 48 GB | No | No | No |
| A100 / H100 80GB | 80 GB | 256K (full) | No | No |
| Mac, 128 GB unified memory | 96 GB | 256K (full) | No | No |
| H200 | 141 GB | 256K (full) | 256K (full) | 256K (full) |
| B200 | 180 GB | 256K (full) | 256K (full) | 256K (full) |
Model details
- Parameters
- 123.6B (123,611,012,096)
- Experts
- 512 routed experts, all loaded
- Active per token
- 12B
- Layers
- 8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache
- Attention cache
- 2 KV heads × 128
- Context length
- 262,144 tokens
- Published weights
- 230 GB (BF16)
- On Hugging Face
- nvidia/
NVIDIA-Nemotron-3-Super-120B-A12B-BF16
Why this estimate looks this way
At Q4_K_M and an 8K-token context, Nemotron 3 Super 120B-A12B uses 69.6 GB for weights, 64 MB for its FP16 KV cache and 7.47 GB for estimated runtime overhead, totaling 77.2 GB. The overhead is a 0.5 GB base plus 10% of weights and cache.
This is a mixture-of-experts model: all 512 experts contribute to the 123.6B parameters held in memory, even though only about 12B parameters run per token. Using only active parameters would understate VRAM.
Its attention layout matters for long context: 8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache. The FP16 cache grows by about 8 KB per additional token per request.
The architecture and published checkpoint size come from the model's config.json and weight files. These are estimates rather than measured peak memory; inference engines can reserve extra buffers or preallocate the full configured cache.
Nemotron 3 Super 120B-A12B next to similar-sized models
The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.
| Model | Parameters | Total | KV per token | Smallest setup |
|---|---|---|---|---|
| Nemotron 3 Super 120B-A12B | 123.6B, 12B active | 77.2 GB | 8 KB | A100 80GB |
| Nemotron 3.5 Super VL 120B-A12B | 124.5B, 12B active | 77.7 GB | 8 KB | A100 80GB |
| Qwen3.5 122B-A10B | 125.1B, 10B active | 78.2 GB | 24 KB | A100 80GB |
| Mistral Medium 3.5 128B | 127.7B | 82.7 GB | 352 KB | Ryzen AI Max+ 395 (128 GB, 96 GB usable) |
Nemotron 3 Super 120B-A12B VRAM questions
How much VRAM do I need to run Nemotron 3 Super 120B-A12B locally?
About 77.2 GB at Q4_K_M with an 8K context and one request, by the bits-per-weight estimate, and 254 GB at BF16. The Q4_K_M GGUF UD-Q4_K_M/
Can I run Nemotron 3 Super 120B-A12B on a 24 GB GPU?
No. The RTX 4090 (24 GB) alone cannot hold it: the smallest listed GGUF, Q2_K at 51.1 GB, needs 56.7 GB with an 8K context.
Which quantization of Nemotron 3 Super 120B-A12B fits in 16 GB of VRAM?
The RTX 5060 Ti 16GB alone cannot hold it: the smallest listed GGUF, Q2_K at 51.1 GB, needs 56.7 GB with an 8K context.
Can I run Nemotron 3 Super 120B-A12B on a Mac?
The M4 Pro Mac (64 GB, 48 GB usable) alone cannot hold it: the smallest listed GGUF, Q2_K at 51.1 GB, needs 56.7 GB with an 8K context. On the M4 Max Mac (128 GB, 96 GB usable), the largest listed GGUF that fits with an 8K context and 0.5 GB free is Q4_K_M (76.9 GB), with its full 256K-token context. By default macOS lets the GPU use about 75% of unified memory.
How much more VRAM does Nemotron 3 Super 120B-A12B need for 32K or 128K tokens of context?
Its FP16 KV cache is 64 MB at 8K, 256 MB at 32K, 1.00 GB at 128K for one request, so 128K adds 960 MB over 8K, plus 10% overhead. A q8_0 cache
Why does Nemotron 3 Super 120B-A12B use more VRAM than its GGUF file size?
The file holds only the weights. At Q4_K_M with an 8K context: 76.9 GB of weights (the file's 82,541,168,480 bytes) + 64 MB of FP16 KV cache + 8.19 GB of compute buffers and runtime (0.5 GB + 10%) = 85.1 GB.
Does Nemotron 3 Super 120B-A12B need less VRAM because only 12B parameters are active?
No. All 512 experts stay loaded, so the Q4_K_M file takes 76.9 GB. The active parameters decide speed: each generated token reads about 6.76 GB of them.
Badge for your model card
Paste it into a Hugging Face or GitHub README; it shows the Q4_K_M total above and links to this page. Q8_0 and FP16 badges.
Numbers read from the model files on Hugging Face on . Compare it with other models in the reproducible model and GPU report.