Ornith 1.5 35B-A3B VRAM requirements

Running Ornith 1.5 35B-A3B with experts in system RAM

llama.cpp's --n-cpu-moe N keeps the experts of the first N layers in RAM. For the measured Q4_K_M GGUF with 32K tokens of context, the smallest N that fits each card, and the speed with 89.6 GB/s of system RAM. Plan the offload for another GPU. The VRAM here can sit below the 23.5 GB of the worked example because it counts the file's measured bytes, keeps the input embedding (token_embd) in system RAM as llama.cpp does and adds a flat 1 GB of buffers, where the estimate above adds 0.5 GB plus 10% to the preset's weights.

In system RAM counts the offloaded experts plus the input embedding (token_embd), which stays in RAM even with --n-cpu-moe 0.

How much VRAM does Ornith 1.5 35B-A3B need?

Ornith 1.5 35B-A3B needs 23.0 GB at Q4_K_M, 39.8 GB at Q8_0 and 74.3 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.

Fine-tuning Ornith 1.5 35B-A3B? Ornith 1.5 35B-A3B VRAM for LoRA, QLoRA and full training.

Run Ornith 1.5 35B-A3B with llama.cpp

llama-server -hf bartowski/Ornith-1.5-35B-A3B-GGUF:Q4_K_M -c 27648 -ngl 99

Ornith-1.5-35B-A3B-Q4_K_M.gguf, 21.9 GB, from bartowski/Ornith-1.5-35B-A3B-GGUF (checked 2026-09-29). On a 24 GB RTX 3090 / 4090, Q4_K_M with -c 27648 takes 23.5 GB and leaves 534 MB.

Ornith 1.5 35B-A3B at 8K, 32K, 128K and 256K (full) tokens of context

10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache. Each extra token of context adds 20 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.

How fast Ornith 1.5 35B-A3B writes

Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth and the parameters read per token. A dash means it does not fit on one card. Try other settings in the speed calculator.

Longest context on one GPU

How many tokens of context Ornith 1.5 35B-A3B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.

Model details

Parameters

36.0B (35,951,822,704)

Experts

256 routed experts, all loaded

Active per token

3B

Layers

10 of its 40 layers use full attention and 30 are linear-attention layers with no growing cache

Attention cache

2 KV heads × 256

Context length

262,144 tokens

Published weights

67.0 GB (BF16)

On Hugging Face

ornith-ai/Ornith-1.5-35B-A3B

Ornith 1.5 35B-A3B next to similar-sized models

The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.