K2-Horizon MoVA 36B-A4B VRAM requirements
K2-Horizon MoVA 36B-A4B has 37.4B parameters, of which about 4B are used per token; all 100 experts still have to be in memory. With an 8K-token context and one request it needs about 25.4 GB of GPU memory at Q4_K_M, 40.5 GB at FP8 and 78.9 GB at FP16/BF16. The published weights take 69.7 GB (BF16). It does not fit on a 24 GB card at Q4_K_M; the smallest setup here that holds it with an 8K context is the RTX 5090 (32 GB).
25.4 GB at Q4_K_M, 8K context, one request
- FP8
- 40.5 GB
- FP16 / BF16
- 78.9 GB
- Published weights
- 69.7 GB
- Smallest setup, Q4_K_M
- RTX 5090 (32 GB)
Worked example: K2-Horizon MoVA 36B-A4B with 32K tokens
Inputs: K2-Horizon MoVA 36B-A4B · Q4_K_M weights · 32,768 tokens of context · one request · FP16 KV cache
- Weights (37.4B at Q4_K_M)
- 21.1 GB
- KV cache (192 KB per token)
- 6.00 GB
- Buffers and runtime (0.5 GB + 10%)
- 3.21 GB
- Total
- 30.3 GB
The smallest setup here that holds it is the RTX 5090 (32 GB), with about 1.69 GB to spare. Change the inputs in the calculator
What makes K2-Horizon MoVA 36B-A4B's memory use different
Per token of context it adds 192 KB of FP16 cache; Granite 4.2 30B (29.3B), the nearest-sized model here with plain full attention, adds 256 KB, so K2-Horizon MoVA 36B-A4B needs 25% less.
As a mixture-of-experts model it reads about 2.25 GB of its 21.1 GB Q4_K_M weights per generated token (11%), so it writes like a much smaller model while needing memory for all of them.
How much VRAM does K2-Horizon MoVA 36B-A4B need?
K2-Horizon MoVA 36B-A4B needs 25.4 GB at Q4_K_M, 42.9 GB at Q8_0 and 78.9 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.
| Precision | Weights | Total | Smallest setup |
|---|---|---|---|
| As published (BF16) | 69.7 GB | 78.9 GB | A100 80GB |
| FP16 / BF16 | 69.7 GB | 78.9 GB | A100 80GB |
| FP8 / INT8 | 34.9 GB | 40.5 GB | M4 Pro Mac (64 GB, 48 GB usable) |
| INT4 (AWQ / GPTQ) | 18.5 GB | 22.5 GB | RTX 3090 (24 GB) |
| GGUF Q8_0 | 37.1 GB | 42.9 GB | M4 Pro Mac (64 GB, 48 GB usable) |
| GGUF Q6_K | 28.6 GB | 33.6 GB | M4 Pro Mac (64 GB, 48 GB usable) |
| GGUF Q5_K_M | 24.7 GB | 29.3 GB | RTX 5090 (32 GB) |
| GGUF Q4_K_M | 21.1 GB | 25.4 GB | RTX 5090 (32 GB) |
| GGUF IQ4_XS | 19.0 GB | 23.0 GB | RTX 3090 (24 GB) |
| GGUF Q3_K_M | 17.0 GB | 20.9 GB | RTX 3090 (24 GB) |
| GGUF IQ3_XXS | 14.4 GB | 18.0 GB | RTX 3090 (24 GB) |
| GGUF Q2_K | 14.6 GB | 18.2 GB | RTX 3090 (24 GB) |
Fine-tuning K2-Horizon MoVA 36B-A4B? K2-Horizon MoVA 36B-A4B VRAM for LoRA, QLoRA and full training.
K2-Horizon MoVA 36B-A4B GGUF files on Hugging Face
The 5 GGUF files llama.cpp picks for K2-Horizon MoVA 36B-A4B with -hf <repo>:<QUANT>, with the bytes the Hugging Face file list reported on 2026-09-29. Each total is that file plus the FP16 KV cache and 0.5 GB + 10%, one request. "vs estimate" compares the file with the bits-per-weight size the table above uses.
| Quant | File | Size | vs estimate | 8K | 32K | 128K |
|---|---|---|---|---|---|---|
| Q4_K_M | IFM/ | 20.8 GB22,368,011,616 bytes | −1% | 25.1 GB | 30.0 GB | 49.8 GB |
| Q5_K_M | IFM/ | 24.6 GB26,439,456,096 bytes | ±0% | 29.2 GB | 34.2 GB | 54.0 GB |
| Q6_K | IFM/ | 28.7 GB30,765,365,856 bytes | ±0% | 33.7 GB | 38.6 GB | 58.4 GB |
| Q8_0 | IFM/ | 37.1 GB39,831,174,496 bytes | ±0% | 43.0 GB | 47.9 GB | 67.7 GB |
| BF16 | IFM/ | 69.8 GB74,924,627,296 bytes | ±0% | 78.9 GB | 83.9 GB | 104 GB |
GB here is GiB (1024³ bytes), as everywhere on this site; Hugging Face shows the Q4_K_M file as 22.4 GB, in 10⁹ bytes.
Which K2-Horizon MoVA 36B-A4B GGUF fits a 16, 24, 32, 48 or 80 GB GPU or a Mac
From the file sizes above: the largest file that fits each machine with an 8K context and 0.5 GB left free, the longest context it then has room for, and the longest context for Q4_K_M (20.8 GB). One request, FP16 KV cache. The Q4_K_M file is 273 MB under its bits-per-weight estimate; the estimate tables and the llama-server command on this page count the estimate, so they show less context.
| Hardware | Largest GGUF, 8K | Its longest context | Q4_K_M, longest context |
|---|---|---|---|
| RTX 5060 Ti 16GB | None fits | — | No |
| RTX 4090 (24 GB) | None fits | — | No |
| RTX 5090 (32 GB) | Q5_K_M, 24.6 GB | 18K | 39K |
| L40S (48 GB) | Q8_0, 37.1 GB | 29K | 116K |
| H100 SXM (80 GB) | BF16, 69.8 GB | 10K | 271K |
| M4 Pro Mac (64 GB, 48 GB usable) | Q8_0, 37.1 GB | 29K | 116K |
| M4 Max Mac (128 GB, 96 GB usable) | BF16, 69.8 GB | 88K | 349K |
Run K2-Horizon MoVA 36B-A4B with llama.cpp
llama-server -hf IFM/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M -c 37888K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf, 22.4 GB, from IFM/
K2-Horizon MoVA 36B-A4B at 8K, 32K, 128K and 512K (full) tokens of context
All 48 layers use full attention. Each extra token of context adds 192 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.
| Context | KV cache, FP16 | Total at Q4_K_M | Total at Q8_0 | Smallest setup, Q4_K_M |
|---|---|---|---|---|
| 8K tokens | 1.50 GB | 25.4 GB | 42.9 GB | RTX 5090 (32 GB) |
| 32K tokens | 6.00 GB | 30.3 GB | 47.9 GB | RTX 5090 (32 GB) |
| 128K tokens | 24.0 GB | 50.1 GB | 67.7 GB | 2× RTX 5090 (64 GB) |
| 512K tokens | 96.0 GB | 129 GB | 147 GB | H200 (141 GB) |
How fast K2-Horizon MoVA 36B-A4B writes
Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth and the parameters read per token. A dash means it does not fit on one card. Try other settings in the speed calculator.
| Hardware | Bandwidth | Q4_K_M | FP8 |
|---|---|---|---|
| RTX 3060 12GB | 360 GB/s | — | — |
| RTX 4090 | 1,008 GB/s | — | — |
| RTX 5090 | 1,792 GB/s | 111–200 | — |
| M4 Max Mac (128 GB) | 546 GB/s | 38–66 | 28–48 |
| M3 Ultra Mac Studio (512 GB) | 819 GB/s | 56–97 | 41–70 |
| H100 SXM | 3,350 GB/s | 181–344 | 141–260 |
| H200 | 4,800 GB/s | 233–459 | 185–352 |
Longest context on one GPU
How many tokens of context K2-Horizon MoVA 36B-A4B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.
| GPU | Memory | Q4_K_M | Q8_0 | FP8 |
|---|---|---|---|---|
| RTX 3060 | 12 GB | No | No | No |
| RTX 4060 Ti 16GB | 16 GB | No | No | No |
| RTX 3090 / 4090 | 24 GB | No | No | No |
| RTX 5090 | 32 GB | 37K | No | No |
| A100 40GB | 40 GB | 76K | No | 2K |
| Mac, 64 GB unified memory | 48 GB | 115K | 30K | 41K |
| L40S / RTX 6000 Ada | 48 GB | 115K | 30K | 41K |
| A100 / H100 80GB | 80 GB | 270K | 185K | 196K |
| Mac, 128 GB unified memory | 96 GB | 348K | 262K | 274K |
| H200 | 141 GB | 512K (full) | 481K | 492K |
| B200 | 180 GB | 512K (full) | 512K (full) | 512K (full) |
Model details
- Parameters
- 37.4B (37,444,792,020)
- Experts
- 100 routed experts, all loaded
- Active per token
- 4B
- Layers
- All 48 layers use full attention
- Attention cache
- 8 KV heads × 128
- Context length
- 524,288 tokens
- Published weights
- 69.7 GB (BF16)
- On Hugging Face
- IFM/
K2-Horizon-MoVA-36B-A4B
Why this estimate looks this way
At Q4_K_M and an 8K-token context, K2-Horizon MoVA 36B-A4B uses 21.1 GB for weights, 1.50 GB for its FP16 KV cache and 2.76 GB for estimated runtime overhead, totaling 25.4 GB. The overhead is a 0.5 GB base plus 10% of weights and cache.
This is a mixture-of-experts model: all 100 experts contribute to the 37.4B parameters held in memory, even though only about 4B parameters run per token. Using only active parameters would understate VRAM.
The architecture and published checkpoint size come from the model's config.json and weight files. These are estimates rather than measured peak memory; inference engines can reserve extra buffers or preallocate the full configured cache.
K2-Horizon MoVA 36B-A4B next to similar-sized models
The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.
| Model | Parameters | Total | KV per token | Smallest setup |
|---|---|---|---|---|
| K2-Horizon MoVA 36B-A4B | 37.4B, 4B active | 25.4 GB | 192 KB | RTX 5090 (32 GB) |
| Ornith 1.5 35B-A3B | 36.0B, 3B active | 23.0 GB | 20 KB | RTX 3090 (24 GB) |
| Qwen3.6 35B-A3B | 36.0B, 3B active | 23.0 GB | 20 KB | RTX 3090 (24 GB) |
| Ornith 1.0 35B | 35.1B, 3B active | 22.4 GB | 20 KB | RTX 3090 (24 GB) |
K2-Horizon MoVA 36B-A4B VRAM questions
How much VRAM do I need to run K2-Horizon MoVA 36B-A4B locally?
About 25.4 GB at Q4_K_M with an 8K context and one request, by the bits-per-weight estimate, and 78.9 GB at BF16. The Q4_K_M GGUF K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf in IFM/
Can I run K2-Horizon MoVA 36B-A4B on a 24 GB GPU?
No. The RTX 4090 (24 GB) alone cannot hold it: the smallest listed GGUF, Q4_K_M at 20.8 GB, needs 25.1 GB with an 8K context.
Which quantization of K2-Horizon MoVA 36B-A4B fits in 16 GB of VRAM?
The RTX 5060 Ti 16GB alone cannot hold it: the smallest listed GGUF, Q4_K_M at 20.8 GB, needs 25.1 GB with an 8K context.
Can I run K2-Horizon MoVA 36B-A4B on a Mac?
On the M4 Pro Mac (64 GB, 48 GB usable), the largest listed GGUF that fits with an 8K context and 0.5 GB free is Q8_0 (37.1 GB), with up to 29K tokens of context. Q4_K_M (20.8 GB) runs with up to 116K tokens of context. On the M4 Max Mac (128 GB, 96 GB usable), the largest listed GGUF that fits with an 8K context and 0.5 GB free is BF16 (69.8 GB), with up to 88K tokens of context. Q4_K_M (20.8 GB) runs with up to 349K tokens of context. By default macOS lets the GPU use about 75% of unified memory.
How much more VRAM does K2-Horizon MoVA 36B-A4B need for 32K or 128K tokens of context?
Its FP16 KV cache is 1.50 GB at 8K, 6.00 GB at 32K, 24.0 GB at 128K for one request, so 128K adds 22.5 GB over 8K, plus 10% overhead. A q8_0 cache
Why does K2-Horizon MoVA 36B-A4B use more VRAM than its GGUF file size?
The file holds only the weights. At Q4_K_M with an 8K context: 20.8 GB of weights (the file's 22,368,011,616 bytes) + 1.50 GB of FP16 KV cache + 2.73 GB of compute buffers and runtime (0.5 GB + 10%) = 25.1 GB.
Does K2-Horizon MoVA 36B-A4B need less VRAM because only 4B parameters are active?
No. All 100 experts stay loaded, so the Q4_K_M file takes 20.8 GB. The active parameters decide speed: each generated token reads about 2.25 GB of them.
Badge for your model card
Paste it into a Hugging Face or GitHub README; it shows the Q4_K_M total above and links to this page. Q8_0 and FP16 badges.
Numbers read from the model files on Hugging Face on . Compare it with other models in the reproducible model and GPU report.