MoE Offload Calculator (--n-cpu-moe)

Use the smallest N that fits: for Qwen3.6-35B-A3B UD-Q4_K_M with 32,768 tokens, an FP16 KV cache and 1 GB of buffers, that is --n-cpu-moe 13, which moves 5.89 GB of experts to system RAM and uses 15.8 GB of VRAM. The file has 18.2 GB of expert tensors and 2.38 GB of everything else, so a 24 GB card holds it whole. A larger N frees more VRAM for context but reads more from slower system RAM.

It keeps the routed-expert weights (the ffn_*_exps tensors) of the first N layers in system RAM and runs them on the CPU. Attention, the router, shared experts and the output head stay on the GPU. It was added to llama.cpp in PR #15077 (August 2025); --cpu-moe does the same for every layer.

Each token uses only a few experts, for example 8 of 256 in Qwen3.6-35B-A3B, so the CPU reads only about 3% of each offloaded layer’s expert bytes per token. The attention and shared weights that every token needs stay on the fast GPU memory.

Yes: layers 0 to N−1, by index, and layers without experts move nothing. GLM-4.7 Flash’s first layer is dense, and in Nemotron 3 Nano 29 of the 52 layers are Mamba or attention layers with no experts, so N counts more layers there than it moves.

The weight bytes are exact. The KV cache follows each model’s config, including hybrid and sliding-window layers. Compute buffers and the CUDA context are a setting (1 GB by default), and speeds are estimates from memory bandwidth: 30–50% of the limit, as llama.cpp reaches with MoE models.