Ling 3.0 Tiny 7.9B-A1.3B VRAM requirements
Ling 3.0 Tiny 7.9B-A1.3B has 7.9B parameters, of which about 1.3B are used per token; all 128 experts still have to be in memory. With an 8K-token context and one request it needs about 5.45 GB of GPU memory at Q4_K_M, 8.64 GB at FP8 and 16.7 GB at FP16/BF16. The published weights take 14.7 GB (BF16). The smallest setup here that holds it at Q4_K_M with an 8K context is the RTX 4060 8GB; on one 24 GB RTX 3090 / 4090 it runs with its full 128K-token context.
5.45 GB at Q4_K_M, 8K context, one request
- FP8
- 8.64 GB
- FP16 / BF16
- 16.7 GB
- Published weights
- 14.7 GB
- Smallest setup, Q4_K_M
- RTX 4060 8GB
Worked example: Ling 3.0 Tiny 7.9B-A1.3B with 32K tokens
Inputs: Ling 3.0 Tiny 7.9B-A1.3B · Q4_K_M weights · 32,768 tokens of context · one request · FP16 KV cache
- Weights (7.9B at Q4_K_M)
- 4.45 GB
- KV cache (7 KB per token)
- 216 MB
- Buffers and runtime (0.5 GB + 10%)
- 989 MB
- Total
- 5.62 GB
The smallest setup here that holds it is the RTX 4060 8GB, with about 2.38 GB to spare. Change the inputs in the calculator
What makes Ling 3.0 Tiny 7.9B-A1.3B's memory use different
It uses multi-head latent attention: each MLA layer caches a latent of 576 values per token instead of 5,120 for 16 full key and value heads, 8.9× less.
18 of its 24 layers keep a fixed-size KDA recurrent state, so the cache grows by 7 KB per token where caching every layer would take 27 KB.
Per token of context it adds 7 KB of FP16 cache; Llama 3.1 8B (8.0B), the nearest-sized model here with plain full attention, adds 128 KB, so Ling 3.0 Tiny 7.9B-A1.3B needs 5% as much.
As a mixture-of-experts model it reads about 750 MB of its 4.45 GB Q4_K_M weights per generated token (16%), so it writes like a much smaller model while needing memory for all of them.
How much VRAM does Ling 3.0 Tiny 7.9B-A1.3B need?
Ling 3.0 Tiny 7.9B-A1.3B needs 5.45 GB at Q4_K_M, 9.15 GB at Q8_0 and 16.7 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.
| Precision | Weights | Total | Smallest setup |
|---|---|---|---|
| As published (BF16) | 14.7 GB | 16.7 GB | RTX 3090 (24 GB) |
| FP16 / BF16 | 14.7 GB | 16.7 GB | RTX 3090 (24 GB) |
| FP8 / INT8 | 7.35 GB | 8.64 GB | RTX 3060 12GB |
| INT4 (AWQ / GPTQ) | 3.91 GB | 4.85 GB | RTX 4060 8GB |
| GGUF Q8_0 | 7.81 GB | 9.15 GB | RTX 3060 12GB |
| GGUF Q6_K | 6.03 GB | 7.19 GB | RTX 4060 8GB |
| GGUF Q5_K_M | 5.21 GB | 6.29 GB | RTX 4060 8GB |
| GGUF Q4_K_M | 4.45 GB | 5.45 GB | RTX 4060 8GB |
| GGUF IQ4_XS | 4.00 GB | 4.96 GB | RTX 4060 8GB |
| GGUF Q3_K_M | 3.59 GB | 4.51 GB | RTX 4060 8GB |
| GGUF IQ3_XXS | 3.03 GB | 3.89 GB | RTX 4060 8GB |
| GGUF Q2_K | 3.08 GB | 3.94 GB | RTX 4060 8GB |
Fine-tuning Ling 3.0 Tiny 7.9B-A1.3B? Ling 3.0 Tiny 7.9B-A1.3B VRAM for LoRA, QLoRA and full training.
Run Ling 3.0 Tiny 7.9B-A1.3B with llama.cpp
llama-server -hf bartowski/Ling-3.0-tiny-GGUF:Q4_K_M -c 131072 -ngl 99Ling-3.0-tiny-Q4_K_M.gguf, 4.9 GB, from bartowski/
Ling 3.0 Tiny 7.9B-A1.3B at 8K, 32K and 128K (full) tokens of context
6 of its 24 layers use multi-head latent attention and 18 are KDA recurrent layers with no growing cache. Each extra token of context adds 7 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.
| Context | KV cache, FP16 | Total at Q4_K_M | Total at Q8_0 | Smallest setup, Q4_K_M |
|---|---|---|---|---|
| 8K tokens | 54 MB | 5.45 GB | 9.15 GB | RTX 4060 8GB |
| 32K tokens | 216 MB | 5.62 GB | 9.32 GB | RTX 4060 8GB |
| 128K tokens | 864 MB | 6.32 GB | 10.0 GB | RTX 4060 8GB |
How fast Ling 3.0 Tiny 7.9B-A1.3B writes
Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth and the parameters read per token. A dash means it does not fit on one card. Try other settings in the speed calculator.
| Hardware | Bandwidth | Q4_K_M | FP8 |
|---|---|---|---|
| RTX 3060 12GB | 360 GB/s | 107–193 | 71–124 |
| RTX 4090 | 1,008 GB/s | 233–460 | 167–313 |
| RTX 5090 | 1,792 GB/s | 326–694 | 249–497 |
| M4 Max Mac (128 GB) | 546 GB/s | 150–279 | 102–183 |
| M3 Ultra Mac Studio (512 GB) | 819 GB/s | 203–391 | 142–262 |
| H100 SXM | 3,350 GB/s | 428–997 | 351–763 |
| H200 | 4,800 GB/s | 480–1,175 | 409–939 |
Longest context on one GPU
How many tokens of context Ling 3.0 Tiny 7.9B-A1.3B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.
| GPU | Memory | Q4_K_M | Q8_0 | FP8 |
|---|---|---|---|---|
| RTX 3060 | 12 GB | 128K (full) | 128K (full) | 128K (full) |
| RTX 4060 Ti 16GB | 16 GB | 128K (full) | 128K (full) | 128K (full) |
| RTX 3090 / 4090 | 24 GB | 128K (full) | 128K (full) | 128K (full) |
| RTX 5090 | 32 GB | 128K (full) | 128K (full) | 128K (full) |
| A100 40GB | 40 GB | 128K (full) | 128K (full) | 128K (full) |
| Mac, 64 GB unified memory | 48 GB | 128K (full) | 128K (full) | 128K (full) |
| L40S / RTX 6000 Ada | 48 GB | 128K (full) | 128K (full) | 128K (full) |
| A100 / H100 80GB | 80 GB | 128K (full) | 128K (full) | 128K (full) |
| Mac, 128 GB unified memory | 96 GB | 128K (full) | 128K (full) | 128K (full) |
| H200 | 141 GB | 128K (full) | 128K (full) | 128K (full) |
| B200 | 180 GB | 128K (full) | 128K (full) | 128K (full) |
Model details
- Parameters
- 7.9B (7,893,392,800)
- Experts
- 128 routed experts, all loaded
- Active per token
- 1.3B
- Layers
- 6 of its 24 layers use multi-head latent attention and 18 are KDA recurrent layers with no growing cache
- Attention cache
- 576 values per token (compressed latent)
- Context length
- 131,072 tokens
- Published weights
- 14.7 GB (BF16)
- On Hugging Face
- inclusionAI/
Ling-3.0-tiny
Why this estimate looks this way
At Q4_K_M and an 8K-token context, Ling 3.0 Tiny 7.9B-A1.3B uses 4.45 GB for weights, 54 MB for its FP16 KV cache and 973 MB for estimated runtime overhead, totaling 5.45 GB. The overhead is a 0.5 GB base plus 10% of weights and cache.
This is a mixture-of-experts model: all 128 experts contribute to the 7.9B parameters held in memory, even though only about 1.3B parameters run per token. Using only active parameters would understate VRAM.
Its attention layout matters for long context: 6 of its 24 layers use multi-head latent attention and 18 are KDA recurrent layers with no growing cache. The FP16 cache grows by about 7 KB per additional token per request.
The 18 KDA layers keep a fixed recurrent state, which this estimate does not include; only the 6 MLA layers add a context-growing KV cache. Real serving memory may be higher.
The architecture and published checkpoint size come from the model's config.json and weight files. These are estimates rather than measured peak memory; inference engines can reserve extra buffers or preallocate the full configured cache.
Ling 3.0 Tiny 7.9B-A1.3B next to similar-sized models
The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.
| Model | Parameters | Total | KV per token | Smallest setup |
|---|---|---|---|---|
| Ling 3.0 Tiny 7.9B-A1.3B | 7.9B, 1.3B active | 5.45 GB | 7 KB | RTX 4060 8GB |
| Gemma 4 E4B | 8.0B | 5.64 GB | 16 KB | RTX 4060 8GB |
| Llama 3.1 8B | 8.0B | 6.58 GB | 128 KB | RTX 4060 8GB |
| Qwen3 8B | 8.2B | 6.81 GB | 144 KB | RTX 4060 8GB |
Badge for your model card
Paste it into a Hugging Face or GitHub README; it shows the Q4_K_M total above and links to this page. Q8_0 and FP16 badges.
Numbers read from the model files on Hugging Face on . Compare it with other models in the reproducible model and GPU report.