Qwen3.8 2.4T-A95B VRAM requirements
Qwen3.8 2.4T-A95B needs about 1,517 GB of VRAM at Q4_K_M with 8K context, more than any GPU setup listed here holds. Qwen3.8 2.4T-A95B has 2.45T parameters, of which about 95B are used per token; all 512 experts still have to be in memory. With the same 8K context and one request it needs 2,507 GB at FP8 and 5,013 GB at FP16/BF16. The published weights take 4,556 GB (BF16). No GPU or multi-GPU group listed here holds it at Q4_K_M with an 8K context.
1,517 GB at Q4_K_M, 8K context, one request
- FP8
- 2,507 GB
- FP16 / BF16
- 5,013 GB
- Published weights
- 4,556 GB
- Smallest setup, Q4_K_M
- None listed
Worked example: Qwen3.8 2.4T-A95B with 32K tokens
Inputs: Qwen3.8 2.4T-A95B · Q4_K_M weights · 32,768 tokens of context · one request · FP16 KV cache
- Weights (2.45T at Q4_K_M)
- 1,378 GB
- KV cache (92 KB per token)
- 2.88 GB
- Buffers and runtime (0.5 GB + 10%)
- 139 GB
- Total
- 1,520 GB
No GPU or group listed here holds it at Q4_K_M with 32K tokens of context. Change the inputs in the calculator
What makes Qwen3.8 2.4T-A95B's memory use different
69 of its 92 layers keep a fixed-size linear-attention state, so the cache grows by 92 KB per token where caching every layer would take 368 KB.
As a mixture-of-experts model it reads about 53.5 GB of its 1,378 GB Q4_K_M weights per generated token (4%), so it writes like a much smaller model while needing memory for all of them.
How much VRAM does Qwen3.8 2.4T-A95B need?
Qwen3.8 2.4T-A95B needs 1,517 GB at Q4_K_M, 2,664 GB at Q8_0 and 5,013 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.
| Precision | Weights | Total | Smallest setup |
|---|---|---|---|
| As published (BF16) | 4,556 GB | 5,013 GB | None listed |
| FP16 / BF16 | 4,556 GB | 5,013 GB | None listed |
| FP8 / INT8 | 2,278 GB | 2,507 GB | None listed |
| INT4 (AWQ / GPTQ) | 1,210 GB | 1,333 GB | None listed |
| GGUF Q8_0 | 2,421 GB | 2,664 GB | None listed |
| GGUF Q6_K | 1,868 GB | 2,056 GB | None listed |
| GGUF Q5_K_M | 1,615 GB | 1,777 GB | None listed |
| GGUF Q4_K_M | 1,378 GB | 1,517 GB | None listed |
| GGUF IQ4_XS | 1,239 GB | 1,364 GB | None listed |
| GGUF Q3_K_M | 1,113 GB | 1,226 GB | None listed |
| GGUF IQ3_XXS | 940 GB | 1,035 GB | 8× H200 (1,128 GB) |
| GGUF Q2_K | 954 GB | 1,051 GB | 8× H200 (1,128 GB) |
Fine-tuning Qwen3.8 2.4T-A95B? Qwen3.8 2.4T-A95B VRAM for LoRA, QLoRA and full training.
Qwen3.8 2.4T-A95B GGUF files on Hugging Face
The 3 GGUF files llama.cpp picks for Qwen3.8 2.4T-A95B with -hf <repo>:<QUANT>, with the bytes the Hugging Face file list reported on 2026-09-29. Each total is that file plus the FP16 KV cache and 0.5 GB + 10%, one request. "vs estimate" compares the file with the bits-per-weight size the table above uses.
| Quant | File | Size | vs estimate | 8K | 32K | 128K |
|---|---|---|---|---|---|---|
| IQ4_XS | unsloth/ | 1,221 GB1,310,876,469,952 bytes | −1% | 1,344 GB | 1,347 GB | 1,356 GB |
| Q8_0 | unsloth/ | 2,422 GB2,600,249,563,264 bytes | ±0% | 2,665 GB | 2,668 GB | 2,677 GB |
| BF16 | unsloth/ | 4,557 GB4,893,172,677,568 bytes | ±0% | 5,014 GB | 5,016 GB | 5,026 GB |
GB here is GiB (1024³ bytes), as everywhere on this site.
Which Qwen3.8 2.4T-A95B GGUF fits a 16, 24, 32, 48 or 80 GB GPU or a Mac
From the file sizes above: the largest file that fits each machine with an 8K context and 0.5 GB left free, the longest context it then has room for, and the longest context for Q4_K_M. One request, FP16 KV cache.
| Hardware | Largest GGUF, 8K | Its longest context | Q4_K_M, longest context |
|---|---|---|---|
| RTX 5060 Ti 16GB | None fits | — | — |
| RTX 4090 (24 GB) | None fits | — | — |
| RTX 5090 (32 GB) | None fits | — | — |
| L40S (48 GB) | None fits | — | — |
| H100 SXM (80 GB) | None fits | — | — |
| M4 Pro Mac (64 GB, 48 GB usable) | None fits | — | — |
| M4 Max Mac (128 GB, 96 GB usable) | None fits | — | — |
Qwen3.8 2.4T-A95B at 8K, 32K, 128K and 256K (full) tokens of context
23 of its 92 layers use full attention and 69 are linear-attention layers with no growing cache. Each extra token of context adds 92 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.
| Context | KV cache, FP16 | Total at Q4_K_M | Total at Q8_0 | Smallest setup, Q4_K_M |
|---|---|---|---|---|
| 8K tokens | 736 MB | 1,517 GB | 2,664 GB | None listed |
| 32K tokens | 2.88 GB | 1,520 GB | 2,666 GB | None listed |
| 128K tokens | 11.5 GB | 1,529 GB | 2,676 GB | None listed |
| 256K tokens | 23.0 GB | 1,542 GB | 2,688 GB | None listed |
Model details
- Parameters
- 2.45T (2,446,182,725,504)
- Experts
- 512 routed experts, all loaded
- Active per token
- 95B
- Layers
- 23 of its 92 layers use full attention and 69 are linear-attention layers with no growing cache
- Attention cache
- 4 KV heads × 256
- Context length
- 262,144 tokens
- Published weights
- 4,556 GB (BF16)
- On Hugging Face
- Qwen/
Qwen3.8-2.4T-A95B - Reported benchmarks
- None published in the model card for GPQA, MMLU-Pro or LiveCodeBench.
Why this estimate looks this way
At Q4_K_M and an 8K-token context, Qwen3.8 2.4T-A95B uses 1,378 GB for weights, 736 MB for its FP16 KV cache and 138 GB for estimated runtime overhead, totaling 1,517 GB. The overhead is a 0.5 GB base plus 10% of weights and cache.
This is a mixture-of-experts model: all 512 experts contribute to the 2.45T parameters held in memory, even though only about 95B parameters run per token. Using only active parameters would understate VRAM.
Its attention layout matters for long context: 23 of its 92 layers use full attention and 69 are linear-attention layers with no growing cache. The FP16 cache grows by about 92 KB per additional token per request.
The architecture and published checkpoint size come from the model's config.json and weight files. These are estimates rather than measured peak memory; inference engines can reserve extra buffers or preallocate the full configured cache.
Qwen3.8 2.4T-A95B next to similar-sized models
The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.
| Model | Parameters | Total | KV per token | Smallest setup |
|---|---|---|---|---|
| Qwen3.8 2.4T-A95B | 2.45T, 95B active | 1,517 GB | 92 KB | None listed |
| Kimi K3 | 2.78T, 104B active | 1,724 GB | 27 KB | None listed |
| DeepSeek V4 Pro | 1.60T, 49B active | 993 GB | 122 KB | 8× H200 (1,128 GB) |
| MiMo V2.6 Pro | 1.02T, 42B active | 636 GB | 50 KB | 8× H100 SXM (640 GB) |
Qwen3.8 2.4T-A95B VRAM questions
How much VRAM do I need to run Qwen3.8 2.4T-A95B locally?
About 1,517 GB at Q4_K_M with an 8K context and one request, by the bits-per-weight estimate, and 5,013 GB at BF16. No single GPU or group listed here holds it at Q4_K_M.
Can I run Qwen3.8 2.4T-A95B on a 24 GB GPU?
No. The RTX 4090 (24 GB) alone cannot hold it: the smallest listed GGUF, IQ4_XS at 1,221 GB, needs 1,344 GB with an 8K context.
Which quantization of Qwen3.8 2.4T-A95B fits in 16 GB of VRAM?
The RTX 5060 Ti 16GB alone cannot hold it: the smallest listed GGUF, IQ4_XS at 1,221 GB, needs 1,344 GB with an 8K context.
Can I run Qwen3.8 2.4T-A95B on a Mac?
The M4 Pro Mac (64 GB, 48 GB usable) alone cannot hold it: the smallest listed GGUF, IQ4_XS at 1,221 GB, needs 1,344 GB with an 8K context. The M4 Max Mac (128 GB, 96 GB usable) alone cannot hold it: the smallest listed GGUF, IQ4_XS at 1,221 GB, needs 1,344 GB with an 8K context. By default macOS lets the GPU use about 75% of unified memory.
How much more VRAM does Qwen3.8 2.4T-A95B need for 32K or 128K tokens of context?
Its FP16 KV cache is 736 MB at 8K, 2.88 GB at 32K, 11.5 GB at 128K for one request, so 128K adds 10.8 GB over 8K, plus 10% overhead. A q8_0 cache
Does Qwen3.8 2.4T-A95B need less VRAM because only 95B parameters are active?
No. All 512 experts stay loaded, so the Q4_K_M weights take 1,378 GB. The active parameters decide speed: each generated token reads about 53.5 GB of them.
Badge for your model card
Paste it into a Hugging Face or GitHub README; it shows the Q4_K_M total above and links to this page. Q8_0 and FP16 badges.
Numbers read from the model files on Hugging Face on . Compare it with other models in the reproducible model and GPU report.