Ornith 1.5 35B-A3B VRAM requirements

Ornith 1.5 35B-A3B needs about 23.0 GB of VRAM at Q4_K_M with 8K context, so it fits a 24 GB card such as the RTX 3090 or the RTX 4090. Ornith 1.5 35B-A3B has 36.0B parameters, of which about 3B are used per token; all 256 experts still have to be in memory. With the same 8K context and one request it needs 37.5 GB at FP8 and 74.3 GB at FP16/BF16. The published weights take 67.0 GB (BF16). The smallest setup here that holds it at Q4_K_M with an 8K context is the RTX 3090 (24 GB), which runs it with up to 33K tokens of context.

23.0 GB at Q4_K_M, 8K context, one request

FP8
37.5 GB
FP16 / BF16
74.3 GB
Published weights
67.0 GB
Smallest setup, Q4_K_M
RTX 3090 (24 GB)
Open Ornith 1.5 35B-A3B in the calculator

Worked example: Ornith 1.5 35B-A3B with 32K tokens

Inputs: Ornith 1.5 35B-A3B · Q4_K_M weights · 32,768 tokens of context · one request · FP16 KV cache

Weights (36.0B at Q4_K_M)
20.3 GB
KV cache (20 KB per token)
640 MB
Buffers and runtime (0.5 GB + 10%)
2.59 GB
Total
23.5 GB

The smallest setup here that holds it is the RTX 3090 (24 GB), with about 542 MB to spare. Change the inputs in the calculator

What makes Ornith 1.5 35B-A3B's memory use different

30 of its 40 layers keep a fixed-size linear-attention state, so the cache grows by 20 KB per token where caching every layer would take 80 KB.

Per token of context it adds 20 KB of FP16 cache; Granite 4.2 30B (29.3B), the nearest-sized model here with plain full attention, adds 256 KB, so Ornith 1.5 35B-A3B needs 8% as much.

As a mixture-of-experts model it reads about 1.69 GB of its 20.3 GB Q4_K_M weights per generated token (8%), so it writes like a much smaller model while needing memory for all of them.

Its architecture and size match Qwen3.6 35B-A3B, so every memory figure on this page is the same for that model.

Running Ornith 1.5 35B-A3B with experts in system RAM

llama.cpp's --n-cpu-moe N keeps the experts of the first N layers in RAM. For the measured Q4_K_M GGUF with 32K tokens of context, the smallest N that fits each card, and the speed with 89.6 GB/s of system RAM. Plan the offload for another GPU. The VRAM here can sit below the 23.5 GB of the worked example because it counts the file's measured bytes, keeps the input embedding (token_embd) in system RAM as llama.cpp does and adds a flat 1 GB of buffers, where the estimate above adds 0.5 GB plus 10% to the preset's weights.

GPU--n-cpu-moeVRAM usedIn system RAMTokens/s
RTX 5060 Ti 16GB 16 GB 13 15.6 GB 6.20 GB 36–62
RTX 4090 24 GB 0 (all on the GPU) 21.6 GB 273 MB 96–171
RTX 5090 32 GB 0 (all on the GPU) 21.6 GB 273 MB 154–286

In system RAM counts the offloaded experts plus the input embedding (token_embd), which stays in RAM even with --n-cpu-moe 0.

How much VRAM does Ornith 1.5 35B-A3B need?

Ornith 1.5 35B-A3B needs 23.0 GB at Q4_K_M, 39.8 GB at Q8_0 and 74.3 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.

PrecisionWeightsTotalSmallest setup
As published (BF16) 67.0 GB 74.3 GB A100 80GB
FP16 / BF16 67.0 GB 74.3 GB A100 80GB
FP8 / INT8 33.5 GB 37.5 GB M4 Pro Mac (64 GB, 48 GB usable)
INT4 (AWQ / GPTQ) 17.8 GB 20.2 GB RTX 3090 (24 GB)
GGUF Q8_0 35.6 GB 39.8 GB M4 Pro Mac (64 GB, 48 GB usable)
GGUF Q6_K 27.5 GB 30.9 GB RTX 5090 (32 GB)
GGUF Q5_K_M 23.7 GB 26.8 GB RTX 5090 (32 GB)
GGUF Q4_K_M 20.3 GB 23.0 GB RTX 3090 (24 GB)
GGUF IQ4_XS 18.2 GB 20.7 GB RTX 3090 (24 GB)
GGUF Q3_K_M 16.4 GB 18.7 GB RTX 3090 (24 GB)
GGUF IQ3_XXS 13.8 GB 15.9 GB RTX 4060 Ti 16GB
GGUF Q2_K 14.0 GB 16.1 GB RTX 3090 (24 GB)

Fine-tuning Ornith 1.5 35B-A3B? Ornith 1.5 35B-A3B VRAM for LoRA, QLoRA and full training.

Ornith 1.5 35B-A3B GGUF files on Hugging Face

The 8 GGUF files llama.cpp picks for Ornith 1.5 35B-A3B with -hf <repo>:<QUANT>, with the bytes the Hugging Face file list reported on 2026-09-29. Each total is that file plus the FP16 KV cache and 0.5 GB + 10%, one request. "vs estimate" compares the file with the bits-per-weight size the table above uses.

QuantFileSizevs estimate 8K32K128K
Q2_K bartowski/Ornith-1.5-35B-A3B-GGUF 12.2 GB13,085,779,616 bytes −13% 14.1 GB14.6 GB16.7 GB
Q3_K_M bartowski/Ornith-1.5-35B-A3B-GGUF 15.6 GB16,696,952,480 bytes −5% 17.8 GB18.3 GB20.4 GB
IQ4_XS bartowski/Ornith-1.5-35B-A3B-GGUF 18.0 GB19,278,554,784 bytes −1% 20.4 GB20.9 GB23.0 GB
Q4_K_M bartowski/Ornith-1.5-35B-A3B-GGUF 20.4 GB21,864,081,056 bytes +1% 23.1 GB23.6 GB25.6 GB
Q5_K_M bartowski/Ornith-1.5-35B-A3B-GGUF 23.7 GB25,492,186,784 bytes ±0% 26.8 GB27.3 GB29.4 GB
Q6_K bartowski/Ornith-1.5-35B-A3B-GGUF 28.4 GB30,526,310,048 bytes +4% 31.9 GB32.5 GB34.5 GB
Q8_0 bartowski/Ornith-1.5-35B-A3B-GGUF 35.2 GB37,813,700,256 bytes −1% 39.4 GB39.9 GB42.0 GB
BF16 bartowski/Ornith-1.5-35B-A3B-GGUF2 parts 66.2 GB71,066,994,208 bytes −1% 73.5 GB74.0 GB76.1 GB

GB here is GiB (1024³ bytes), as everywhere on this site; Hugging Face shows the Q4_K_M file as 21.9 GB, in 10⁹ bytes.

Which Ornith 1.5 35B-A3B GGUF fits a 16, 24, 32, 48 or 80 GB GPU or a Mac

From the file sizes above: the largest file that fits each machine with an 8K context and 0.5 GB left free, the longest context it then has room for, and the longest context for Q4_K_M (20.4 GB). One request, FP16 KV cache. Where nothing fits, the --n-cpu-moe table above keeps experts in system RAM.

HardwareLargest GGUF, 8KIts longest contextQ4_K_M, longest context
RTX 5060 Ti 16GB Q2_K, 12.2 GB 73K No
RTX 4090 (24 GB) Q4_K_M, 20.4 GB 27K 27K
RTX 5090 (32 GB) Q5_K_M, 23.7 GB 227K 256K (full)
L40S (48 GB) Q8_0, 35.2 GB 256K (full) 256K (full)
H100 SXM (80 GB) BF16, 66.2 GB 256K (full) 256K (full)
M4 Pro Mac (64 GB, 48 GB usable) Q8_0, 35.2 GB 256K (full) 256K (full)
M4 Max Mac (128 GB, 96 GB usable) BF16, 66.2 GB 256K (full) 256K (full)

Run Ornith 1.5 35B-A3B with llama.cpp

llama-server -hf bartowski/Ornith-1.5-35B-A3B-GGUF:Q4_K_M -c 27648

Ornith-1.5-35B-A3B-Q4_K_M.gguf, 21.9 GB, from bartowski/Ornith-1.5-35B-A3B-GGUF (checked 2026-09-29). On a 24 GB RTX 3090 / 4090, Q4_K_M with -c 27648 takes 23.5 GB and leaves 534 MB.

Ornith 1.5 35B-A3B at 8K, 32K, 128K and 256K (full) tokens of context

10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache. Each extra token of context adds 20 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.

ContextKV cache, FP16Total at Q4_K_MTotal at Q8_0Smallest setup, Q4_K_M
8K tokens 160 MB 23.0 GB 39.8 GB RTX 3090 (24 GB)
32K tokens 640 MB 23.5 GB 40.3 GB RTX 3090 (24 GB)
128K tokens 2.50 GB 25.5 GB 42.4 GB RTX 5090 (32 GB)
256K tokens 5.00 GB 28.3 GB 45.1 GB RTX 5090 (32 GB)

How fast Ornith 1.5 35B-A3B writes

Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth and the parameters read per token. A dash means it does not fit on one card. Try other settings in the speed calculator.

HardwareBandwidthQ4_K_MFP8
RTX 3060 12GB 360 GB/s ——
RTX 4090 1,008 GB/s 124–226—
RTX 5090 1,792 GB/s 193–369—
M4 Max Mac (128 GB) 546 GB/s 74–12948–83
M3 Ultra Mac Studio (512 GB) 819 GB/s 104–18769–121
H100 SXM 3,350 GB/s 288–594215–418
H200 4,800 GB/s 348–754270–549

Longest context on one GPU

How many tokens of context Ornith 1.5 35B-A3B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.

GPUMemoryQ4_K_MQ8_0FP8
RTX 3060 12 GB NoNoNo
RTX 4060 Ti 16GB 16 GB NoNoNo
RTX 3090 / 4090 24 GB 33KNoNo
RTX 5090 32 GB 256K (full)NoNo
A100 40GB 40 GB 256K (full)No100K
Mac, 64 GB unified memory 48 GB 256K (full)256K (full)256K (full)
L40S / RTX 6000 Ada 48 GB 256K (full)256K (full)256K (full)
A100 / H100 80GB 80 GB 256K (full)256K (full)256K (full)
Mac, 128 GB unified memory 96 GB 256K (full)256K (full)256K (full)
H200 141 GB 256K (full)256K (full)256K (full)
B200 180 GB 256K (full)256K (full)256K (full)

Model details

Parameters
36.0B (35,951,822,704)
Experts
256 routed experts, all loaded
Active per token
3B
Layers
10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache
Attention cache
2 KV heads × 256
Context length
262,144 tokens
Published weights
67.0 GB (BF16)
On Hugging Face
ornith-ai/Ornith-1.5-35B-A3B
Reported benchmarks
GPQA Diamond 89.2. Vendor-reported: from the maker’s own model card, not rerun here; read 2026-09-30.

Why this estimate looks this way

At Q4_K_M and an 8K-token context, Ornith 1.5 35B-A3B uses 20.3 GB for weights, 160 MB for its FP16 KV cache and 2.54 GB for estimated runtime overhead, totaling 23.0 GB. The overhead is a 0.5 GB base plus 10% of weights and cache.

This is a mixture-of-experts model: all 256 experts contribute to the 36.0B parameters held in memory, even though only about 3B parameters run per token. Using only active parameters would understate VRAM.

Its attention layout matters for long context: 10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache. The FP16 cache grows by about 20 KB per additional token per request.

The architecture and published checkpoint size come from the model's config.json and weight files. These are estimates rather than measured peak memory; inference engines can reserve extra buffers or preallocate the full configured cache.

Ornith 1.5 35B-A3B next to similar-sized models

The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.

ModelParametersTotalKV per tokenSmallest setup
Ornith 1.5 35B-A3B 36.0B, 3B active 23.0 GB 20 KB RTX 3090 (24 GB)
Qwen3.6 35B-A3B 36.0B, 3B active 23.0 GB 20 KB RTX 3090 (24 GB)
Ornith 1.0 35B 35.1B, 3B active 22.4 GB 20 KB RTX 3090 (24 GB)
K2-Horizon MoVA 36B-A4B 37.4B, 4B active 25.4 GB 192 KB RTX 5090 (32 GB)

Ornith 1.5 35B-A3B VRAM questions

How much VRAM do I need to run Ornith 1.5 35B-A3B locally?

About 23.0 GB at Q4_K_M with an 8K context and one request, by the bits-per-weight estimate, and 74.3 GB at BF16. The Q4_K_M GGUF Ornith-1.5-35B-A3B-Q4_K_M.gguf in bartowski/Ornith-1.5-35B-A3B-GGUF is 21,864,081,056 bytes (20.4 GB); with an 8K context it needs 23.1 GB. The smallest setup listed here that holds Q4_K_M is the RTX 3090 (24 GB).

Can I run Ornith 1.5 35B-A3B on a 24 GB GPU?

Yes. On the RTX 4090 (24 GB), the largest listed GGUF that fits with an 8K context and 0.5 GB free is Q4_K_M (20.4 GB), with up to 27K tokens of context.

Which quantization of Ornith 1.5 35B-A3B fits in 16 GB of VRAM?

On the RTX 5060 Ti 16GB, the largest listed GGUF that fits with an 8K context and 0.5 GB free is Q2_K (12.2 GB), with up to 73K tokens of context.

Can I run Ornith 1.5 35B-A3B on a Mac?

On the M4 Pro Mac (64 GB, 48 GB usable), the largest listed GGUF that fits with an 8K context and 0.5 GB free is Q8_0 (35.2 GB), with its full 256K-token context. On the M4 Max Mac (128 GB, 96 GB usable), the largest listed GGUF that fits with an 8K context and 0.5 GB free is BF16 (66.2 GB), with its full 256K-token context. By default macOS lets the GPU use about 75% of unified memory.

How much more VRAM does Ornith 1.5 35B-A3B need for 32K or 128K tokens of context?

Its FP16 KV cache is 160 MB at 8K, 640 MB at 32K, 2.50 GB at 128K for one request, so 128K adds 2.34 GB over 8K, plus 10% overhead. A q8_0 cache (-ctk q8_0 -ctv q8_0) takes 1.33 GB at 128K. Its 30 linear-attention layers keep a fixed-size state, so only 10 of 40 layers add cache.

Why does Ornith 1.5 35B-A3B use more VRAM than its GGUF file size?

The file holds only the weights. At Q4_K_M with an 8K context: 20.4 GB of weights (the file's 21,864,081,056 bytes) + 160 MB of FP16 KV cache + 2.55 GB of compute buffers and runtime (0.5 GB + 10%) = 23.1 GB.

Does Ornith 1.5 35B-A3B need less VRAM because only 3B parameters are active?

No. All 256 experts stay loaded, so the Q4_K_M file takes 20.4 GB. The active parameters decide speed: each generated token reads about 1.69 GB of them.

Badge for your model card

Paste it into a Hugging Face or GitHub README; it shows the Q4_K_M total above and links to this page. Q8_0 and FP16 badges.

Ornith 1.5 35B-A3B VRAM: 23.0 GB at Q4_K_M, 8K context

Numbers read from the model files on Hugging Face on . Compare it with other models in the reproducible model and GPU report.