Fine-Tuning VRAM Calculator

Gradient checkpointing

VRAM for common setups

2,048 tokens, micro-batch 1, gradient checkpointing on, LoRA rank 16 on all linear layers, AdamW; full fine-tuning in mixed precision. Each figure opens the calculator with that setup.

Checked against published numbers

15 public measurements with their settings, next to this calculator’s estimate for the same setup. The median error is 1.6% and the largest 7.5%. The activation constants and runtime costs were fitted to the Unsloth-blog rows (both frameworks), so those match partly by construction; the three PEFT and Giles Thomas rows were not used for fitting.

“Published” is the card size for the longest-sequence rows and PyTorch’s peak for the others; the estimate is compared like for like (without the 0.5 GB CUDA context for PyTorch peaks).

What is not checked

We found no public measurement with its settings for LoRA on a 16-bit base with gradient checkpointing, Unsloth without gradient checkpointing, MoE models, or micro-batches above 4, so those estimates combine checked parts but are not checked themselves. Unsloth’s longest-sequence tables grow exactly linearly with the card size, so Unsloth partly extrapolated them. Full fine-tuning in mixed precision has one check, 8% low, because DeepSpeed keeps an extra FP16 copy of the weights.

Next to rule-of-thumb tables

Unsloth’s requirements page lists an “absolute minimum” per model size without the sequence length, rank or batch. It is guidance, not a measurement, so it is not in the error figures. The estimate column is Unsloth at 512 tokens, rank 16, batch 1.

Unsloth’s own benchmark implies more than its table for 8B: at rank 32 it fits 2,972 tokens on an 8 GB card, and its figures grow in a straight line that puts a very short sample at about 7.4 GB. LLaMA-Factory’s table (marked “estimated”) gives a 7B model 120 GB for full fine-tuning in 32 bits, 60 GB in pure 16-bit, 16 GB for LoRA and 6 GB for 4-bit QLoRA; for Llama 3.1 8B at 2,048 tokens with Transformers this calculator gives 125 GB, 65.2 GB, 21.3 GB and 14.8 GB.

How it is calculated

• Weights. LoRA: 2 bytes per parameter. QLoRA: every linear layer except the output layer in bitsandbytes NF4, 4 bits plus an 8-bit scale per 64 weights and an FP32 scale per 256 blocks with double quantization (4.127 bits; 4.5 without); embeddings, output layer and norms stay BF16 in Unsloth and become FP32 in Transformers, because prepare_model_for_kbit_training casts them.
• LoRA parameters. r × (d_in + d_out) for each adapted matrix in each layer, from the config’s hidden size, head counts and feed-forward width (all experts in an MoE layer). Llama 3.1 8B at r = 16 on all linear layers: 41,943,040. Adapters and their gradients are FP32 (4 + 4 bytes); AdamW adds 8 bytes, 8-bit AdamW 2.
• Full fine-tuning. Mixed precision: 16 bytes per parameter with AdamW (BF16 weights 2, FP32 master copy 4, gradients 2, m and v 8, as counted in the ZeRO paper), 10 with 8-bit AdamW. Pure BF16: 8 (PyTorch’s AdamW keeps m and v in the weights’ dtype), 6 with 8-bit AdamW.
• Activations. A layer saves about 10h + 2q + 4kv + 8f bytes per token in BF16 with flash attention (h hidden size; q and kv the query and key/value widths; f the feed-forward width, for MoE the experts a token uses), plus a BF16 copy of each adapted layer’s input. Without checkpointing every layer keeps this. Transformers’ checkpointing keeps each layer’s input (2h bytes) plus 2.6 times one layer while it is recomputed; Unsloth’s offloaded checkpointing keeps 0.92 times one layer on the GPU and the inputs in system RAM.
• Logits. Transformers computes the loss on FP32 logits: about 14 bytes per vocabulary entry per token, 1.8 MB for Llama 3’s 128,256 entries. Unsloth’s fused cross-entropy (Apple’s Cut Cross Entropy) never builds them.
• Runtime. 0.6 GB (Transformers) or 1.2 GB (Unsloth) for the CUDA context, cuBLAS workspace and allocator. QLoRA adds a dequantized copy of the largest matrix, and Transformers QLoRA a BF16 copy of the FP32 output layer.
• Not modelled. Several GPUs (FSDP, DeepSpeed ZeRO), CPU offload of weights or optimizer, evaluation or generation during training, and DPO or GRPO (reference model, rollouts).

Sources: bitsandbytes and the QLoRA paper (NF4, double quantization), PEFT’s prepare_model_for_kbit_training, the ZeRO paper (16 bytes per parameter), Korthikanti et al. (activation memory), Unsloth on offloaded checkpointing and Cut Cross Entropy.

Sources read on 2026-09-29.

To run the model after training, see the LLM VRAM calculator. How the site’s inference estimates compare with real runs: predicted vs measured.