Nemotron 3.5 Super VL 120B-A12B VRAM requirements

How much VRAM does Nemotron 3.5 Super VL 120B-A12B need?

Nemotron 3.5 Super VL 120B-A12B needs 77.7 GB at Q4_K_M, 136 GB at Q8_0 and 256 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.

Fine-tuning Nemotron 3.5 Super VL 120B-A12B? Nemotron 3.5 Super VL 120B-A12B VRAM for LoRA, QLoRA and full training.

Nemotron 3.5 Super VL: BF16, NVFP4 and visual inputs

The NVIDIA model card describes a 120B-parameter vision-language MoE with 12B active per token, a C-RADIO vision encoder and a shared MTP head. The published BF16 package is 248,913,061,416 bytes (232 GB); its safetensors metadata counts 124.5B parameters across the checkpoint.

The official NVFP4 package has 88 weight shards totaling 75,010,188,240 bytes (69.9 GB), before KV cache and runtime buffers. Its packed-tensor count is not the original parameter count. Use this file total when planning that package rather than multiplying 120B by four bits.

NVIDIA’s BF16 serving examples use four H100 GPUs or two GB200 GPUs, with vLLM 0.32.0. The command below starts with one request and a 32K context. Image patches, video frames, prompt text and generated output share the configured context budget; processing visual inputs also uses working memory outside the KV cache.

vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 --served-model-name nemotron-3.5-super-vl --tensor-parallel-size 4 --trust-remote-code --max-model-len 32768 --max-num-seqs 1 --gpu-memory-utilization 0.85 --reasoning-parser nemotron_v3

The released configuration has 88 blocks: 40 Mamba-2, 40 MoE and 8 attention blocks. With 2 KV heads of 128 values, those eight attention blocks add 8.00 GB of FP16 cache per request at 1M tokens. Fixed recurrent state and visual processing remain separate. ModelVRAM has not run this model on a GPU. The model card identifies its license as OpenMDW 1.1.

Nemotron 3.5 Super VL 120B-A12B at 8K, 32K, 128K and 1M (full) tokens of context

8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache. Each extra token of context adds 8 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.

How fast Nemotron 3.5 Super VL 120B-A12B writes

Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth and the parameters read per token. A dash means it does not fit on one card. Try other settings in the speed calculator.

Longest context on one GPU

How many tokens of context Nemotron 3.5 Super VL 120B-A12B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.

Model details

Parameters

124.5B (124,453,395,976); NVIDIA describes 120B total and 12B active per token; this count includes the vision-language checkpoint as reported by safetensors metadata

Experts

512 routed experts, all loaded

Active per token

12B

Layers

8 of its 88 layers use full attention and 80 are Mamba or feed-forward layers with no growing cache

Attention cache

2 KV heads × 128

Context length

1,048,576 tokens

Published weights

232 GB (BF16)

On Hugging Face

nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16

Reported benchmarks

None published in the model card for GPQA, MMLU-Pro or LiveCodeBench.

Nemotron 3.5 Super VL 120B-A12B next to similar-sized models

The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.