Ling 3.1 Flash 560B-A25B VRAM requirements

How much VRAM does Ling 3.1 Flash 560B-A25B need?

Ling 3.1 Flash 560B-A25B needs 355 GB at Q4_K_M, 623 GB at Q8_0 and 1,173 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.

Fine-tuning Ling 3.1 Flash 560B-A25B? Ling 3.1 Flash 560B-A25B VRAM for LoRA, QLoRA and full training.

Run Ling-3.1-flash with SGLang

The official MIT-licensed checkpoint stores its weights mainly in FP8: 573,781,078,249 bytes (534 GB). The publisher describes a 560B model with 25B active parameters per token. Those active parameters reduce computation; all 512 routed experts still need storage. The safetensors metadata reports 572.1B stored parameters, which is the count used for the alternative-precision estimates above.

The model card recommends SGLang’s day-0 image with 8 H200 GPUs, TP8 and EP8, and MTP speculative decoding. Its alternative is 4 GB300 GPUs with --tp 4 --ep 4. These are the publisher’s serving examples, not minimum-memory claims.

docker run --gpus all --network host --ipc=host --rm lmsysorg/sglang:dev-ling-3.1-flash python3 -m sglang.launch_server --model-path inclusionAI/Ling-3.1-flash --tp 8 --ep 8 --speculative-algorithm NEXTN --host 0.0.0.0 --port 30000

The released configuration sets 262,144 tokens, while the model card advertises a 1M-token context. This page uses 262,144. Seven KDA layers followed by one MLA layer repeat nine times; only the nine MLA layers add cache with context. Their compressed cache has 512 latent values plus 64 rotary values per token. KDA recurrent state, MTP and temporary serving allocations still need runtime headroom.

ModelVRAM has not run this command on a GPU. The quantization rows are storage estimates; check runtime support and actual files before choosing a quantized deployment.

Ling 3.1 Flash 560B-A25B at 8K, 32K, 128K and 256K (full) tokens of context

9 of its 72 layers use multi-head latent attention and 63 are KDA recurrent layers with no growing cache. Each extra token of context adds 10 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.

How fast Ling 3.1 Flash 560B-A25B writes

Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth and the parameters read per token. A dash means it does not fit on one card. Try other settings in the speed calculator.

Longest context on a multi-GPU group

No single card in this table holds it at Q4_K_M, Q8_0 or FP8, so these rows use GPU groups: how many tokens of context Ling 3.1 Flash 560B-A25B fits on each, in one tensor-parallel group with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.

Model details

Parameters

572.1B (572,120,680,768); the model card describes 560B total and 25B active per token; this count comes from the published safetensors metadata

Experts

512 routed experts, all loaded

Active per token

25B

Layers

9 of its 72 layers use multi-head latent attention and 63 are KDA recurrent layers with no growing cache

Attention cache

576 values per token (compressed latent)

Context length

262,144 tokens

Published weights

534 GB (FP8)

On Hugging Face

inclusionAI/Ling-3.1-flash

Reported benchmarks

None published in the model card for GPQA, MMLU-Pro or LiveCodeBench.

Ling 3.1 Flash 560B-A25B next to similar-sized models

The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.