Why a 24 GB GPU doesn't give you 24 GB for an LLM: CUDA context, display, Windows WDDM and runtime buffers

What CUDA reports before a model loads

llama.cpp prints two numbers per GPU. Device 0: …, VRAM: N MiB is the card's total as CUDA reports it (totalGlobalMem, ggml-cuda.cu:339); using device CUDA0 (…) - N MiB free is what cudaMemGetInfo says is free when the model starts loading (ggml-cuda.cu:5056, src/llama.cpp:303-308). The difference is whatever the driver, the desktop and other programs already hold on that card. From public reports:

On Linux CUDA already reports a total a few hundred MiB below the size on the box (24,126 MiB for a 3090, not 24,576), and in these logs 713–996 MiB in all was unavailable. On Windows the total matches the box (12,287 MiB for a 3060) but 1,033–1,323 MiB was taken before loading, also on cards that were the second GPU. None of the reports says whether a monitor was attached or what else ran, so no figure for the desktop alone is given here.

llama.cpp's own buffers

After the weights and the KV cache, llama.cpp reserves a compute buffer on every device, sized for a worst-case batch of min(n_ctx, n_ubatch) tokens (llama-context.cpp:627-628, 671-674) and logged per device (733-737). The defaults are -b 2048 and -ub 512 (common/common.h:453-454); -ub is the setting that changes the reserved batch (arg.cpp:1673-1677). Read at llama.cpp 19e28a2.

• Gemma 4 31B Q5_K_M, -c 128000 -b 512 -ub 256, RTX 5060 Ti + RTX 3060: 218.04 MiB and 261.25 MiB per GPU, plus 539.04 MiB on the host.
• Qwen3.6 27B Q4_K_M, --ctx-size 131072, RTX 2080 Ti + Tesla P100: 668.03 MiB and 523.54 MiB per GPU.

With --fit (on by default) llama-server sizes context and layers so that --fit-target, 1,024 MiB per device by default (common.h), stays free after its own measurement; the --fit preview replays that with this site's estimate.

vLLM: --gpu-memory-utilization

gpu_memory_utilization is “the fraction of GPU memory to be used for the model executor”, default 0.92, and it is per instance (vllm/config/cache.py:103-111). At startup vLLM requests ceil(total × utilization) of the total memory and stops with “Free memory on device … is less than desired GPU memory utilization” if less than that is free (v1/worker/utils.py:539-558). Of the requested share, what the weights, the activation peak, non-torch memory and the CUDA graph estimate leave is the KV cache (gpu_worker.py:673-695). Read at ac7f3e1 (v0.30.0 logic).

Totals are the ones the vLLM memory calculator uses, read from startup logs where one exists (the RTX 3090's 23.56 GiB is vLLM #15877).

What this site assumes, and where it differs

• Each card counts at the size it is sold with (24 GB for a 3090), read as GiB.
• Every GPU pays 0.5 GB for the CUDA context and runtime; 10% of weights and KV cache is added for compute buffers and fragmentation; a setting counts as a fit only with 0.5 GB left. On a 24 GB card that is 20.9 GB of weights and cache, 23.5 GB in all.
• The desktop, other programs and Windows' share are not modelled.

That 23.5 GB limit is above the free memory in every log above, by 0.20 GB to 1.68 GB. What keeps the site's fits honest is that its totals run high: on the accuracy page whole-card totals came out 5.6%–8.1% above measured use (4 runs, one RTX 4090 report). If that holds, a setting right at the limit would really use:

The RTX 3090 Ti log, 2.2 GB short of its size with nothing said about what else ran, is the exception. On Windows the margin left in these logs is only 0.03 GB–0.14 GB, and one calibration report is thin evidence, so for a setting the calculator marks close to the limit, close the browser and other GPU programs first, or pick the next smaller context. The calculator shows the total; on several cards each one pays the runtime share (splitting a model across GPUs), and a quantized cache frees context room (KV cache quantization).