How to split an LLM across two or more GPUs: VRAM per card in llama.cpp and vLLM

llama.cpp: the split modes

Read at master 19e28a2 (2026-09-29). From the help text of --split-mode (common/arg.cpp:2803-2826):

• none: one GPU only, the one --main-gpu names; the others are dropped (src/llama.cpp:288-301).
• layer (default): “split layers and KV across GPUs (pipelined)”.
• row: “split weight across GPUs by rows (parallelized)”; the weights go into a split buffer across the devices (src/llama-model.cpp:1136-1162), and --main-gpu is the GPU “for intermediate results and KV” (arg.cpp:2854-2862).
• tensor: “split weights and KV across GPUs (parallelized, EXPERIMENTAL)”, and only for some architectures.

What each GPU holds in layer mode

1. Split proportions. --tensor-split takes one proportion per GPU, e.g. 3,1 (arg.cpp:2827-2853). Without it, each GPU's free memory at load time is used; either way the proportions are summed and normalised to cut points (llama-model.cpp:1553-1584).
2. Layers. With every layer on a GPU there are layers + 1 slots, the last one being the output layer, and slot i goes to the first GPU whose cut point is above i ÷ (layers + 1) (1586-1598). Two equal cards and Llama 3.1 70B's 80 layers: 41 and 39 layers, the output layer on the second.
3. Embeddings. The input (token embedding) layer always stays on the CPU (1600-1602); the output layer goes to the last GPU (1610-1611). A model with tied embeddings such as Gemma 4 loads a duplicate of the embedding as its output matrix (src/models/gemma4.cpp:49-53), so that matrix is in system RAM and on the last GPU.
4. KV cache. Each layer's K and V are allocated on that layer's device, unless the cache is kept on the CPU (src/llama-kv-cache.cpp:214-220), so the cache splits in the same proportion as the layers.
5. Buffers. Every device gets its own compute buffer, logged per device (src/llama-context.cpp:733-737). When several GPUs hold every layer and the KV cache in layer mode, llama.cpp can turn on pipeline parallelism (if every GPU backend supports it), which its source notes “increases memory usage” (427-434).

vLLM: tensor and pipeline parallelism

Read at main ac7f3e1, the code the site's vLLM calculator follows (v0.30.0 has the same logic).

• --tensor-parallel-size N: every linear layer is cut into N equal parts, its output or input dimension divided by N (linear.py:494, 1681), so each GPU holds 1/N of the weights. The attention heads must divide by N (config/model.py:1502-1509), and each GPU caches max(1, KV heads ÷ N) KV heads, copying heads when there are fewer than GPUs (1608-1628). Because every GPU gets the same share, the smallest card sets the limit.
• --gpu-memory-utilization (default 0.92) is the fraction of each GPU's memory vLLM takes; whatever the weights and activations leave of it becomes KV cache (config/cache.py:103-111).
• --pipeline-parallel-size N: the layers are divided evenly between N stages, any remainder going to the stages before the last, and the VLLM_PP_LAYER_PARTITION environment variable sets the layer counts by hand (distributed/utils.py:128-167). Each stage holds its layers' weights and KV cache, so with that variable a smaller card can be given fewer layers.

Per GPU with the vLLM memory calculator's estimate, AWQ 4-bit weights, --max-model-len 32768, tensor parallel over the group: