LightOnOCR-3 4B VRAM requirements

How much VRAM does LightOnOCR-3 4B need?

LightOnOCR-3 4B needs 3.59 GB at Q4_K_M, 5.72 GB at Q8_0 and 10.1 GB at FP16 with 8,192 tokens of context and one request. Each total below is the weights plus the FP16 KV cache and the runtime overhead (0.5 GB plus 10%), and each row opens the calculator with that setting. What the GGUF names mean.

Fine-tuning LightOnOCR-3 4B? LightOnOCR-3 4B VRAM for LoRA, QLoRA and full training.

LightOnOCR-3 4B GGUF files, mmproj and memory per page

LightOnOCR-3 4B is LightOn’s OCR model, under Apache 2.0: given a page image and an empty prompt it writes the page as Markdown, and with the prompt grounding it puts a label and a bounding box before every block. A page costs little cache because only 8 of the 32 layers keep one. A US Letter page rendered at 2048 px on the long side, the size LightOn’s client uses for the 4B, becomes 1568 × 2048 pixels and 3,136 image tokens; with the 1,158 output tokens of an average plain transcription that is 4,294 tokens and 134 MB of FP16 cache, 143 MB with grounding’s 1,433. At 1540 px the image is 1,776 tokens; at the 5-megapixel cap behind LightOn’s best olmOCR-Bench score, about 4,800. Each page in flight also holds the 24 linear-attention layers’ fixed state, 48 MB in FP32.

LightOn publishes only the BF16 safetensors (9,078,620,536 bytes, vision encoder included). The GGUF files below are community conversions made with stock llama.cpp, which converts this architecture’s vision encoder too; ModelVRAM has not checked their OCR output. Each file holds the language model, and OCR also needs the vision projector (mmproj). llama.cpp v0.6.0’s -hf downloads it from the same repo, picking the one closest in bits to the model file: the 366,894,656-byte (350 MB) Q8_0 mmproj for every file up to Q8_0 and the 672,423,488-byte (641 MB) F16 one for F16. mradermacher’s i1 repo, the most downloaded, has no mmproj: take one from the static repo and pass it with --mmproj.

LightOn measured up to 3.36 pages per second for the 4B on one H100 with vLLM 0.30.0 and pages rendered at 1540 px (LightOn’s blog, 2026-10-08). For a smaller card, LightOnOCR-3-0.8B (1,706,030,528 bytes in BF16, the same architecture) scores 85.5 on olmOCR-Bench against the 4B’s 86.3, and LightOnOCR-3-1B (2,011,367,464 bytes, the LightOnOCR-2 architecture) 84.5. The 4B’s GGUF files, with the totals this page’s method gives:

The totals leave out the mmproj: add 350 MB for the Q8_0 projector or 641 MB for F16. prithivMLmods/LightOnOCR-3-4B-GGUF has the same Q3_K_M to Q8_0 files, each 448 bytes smaller, BF16 in place of F16, the same Q8_0 mmproj and a 675,569,216-byte BF16 one. File sizes from the Hugging Face file lists, read on 2026-10-12; totals as in the table above, one request, FP16 KV cache.

Run LightOnOCR-3 4B with vLLM or llama.cpp

LightOn serves the model with vLLM and these flags, from the LightOnOCR client’s README, which pins vLLM 0.30.0 with transformers 5.16.1. 18,000 tokens leave room for the largest image the processor accepts (16,384 tokens) and its output, 563 MB of cache for one request. Without --max-model-len, vLLM plans for the configured 262,144 tokens, 8.00 GB of cache for one request: more than a 12 or 16 GB card has left after the 8.46 GB of BF16 weights.

vllm serve lightonai/LightOnOCR-3-4B --limit-mm-per-prompt '{"image": 1}' --max-model-len 18000 --default-chat-template-kwargs '{"enable_thinking": false}' --mm-processor-cache-gb 0 --no-enable-prefix-caching --port 8010

On a smaller card, llama.cpp runs the Q4_K_M file with its Q8_0 mmproj. With the same 18,000 tokens that comes to 4.22 GB by this page’s method, which leaves 3.78 GB of an 8 GB card for the vision encoder’s working memory while it reads a page.

llama-server -hf mradermacher/LightOnOCR-3-4B-GGUF:Q4_K_M -c 18000 --temp 0.1

Send one page image per request, with no text for plain Markdown or the word grounding for blocks with boxes; vLLM batches concurrent requests. LightOn recommends temperature 0.1 rather than greedy decoding. ModelVRAM has not run either command on a GPU.

LightOnOCR-3 4B at 8K, 32K, 128K and 256K (full) tokens of context

8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache. Each extra token of context adds 32 KB of FP16 cache per request. The last column is the smallest setup here, one card or a group, that holds it at Q4_K_M. How the KV cache works.

How fast LightOnOCR-3 4B writes

Tokens per second for one request with 8,192 tokens of context, estimated from memory bandwidth. A dash means it does not fit on one card. Try other settings in the speed calculator.

Longest context on one GPU

How many tokens of context LightOnOCR-3 4B fits on a single card with one request, an FP16 KV cache and 0.5 GB left free. "Full" means the model's whole context window fits.

Model details

Parameters

4.5B (4,539,265,536); LightOn's OCR model on the Qwen3.5-4B vision-language architecture; 333,514,240 of them are the vision encoder

Layers

8 of its 32 layers use full attention and 24 are linear-attention layers with no growing cache

Attention cache

4 KV heads × 256

Context length

262,144 tokens

Published weights

8.46 GB (BF16)

On Hugging Face

lightonai/LightOnOCR-3-4B

Reported benchmarks

None published in the model card for GPQA, MMLU-Pro or LiveCodeBench.

LightOnOCR-3 4B next to similar-sized models

The three models here closest to it in parameter count, at Q4_K_M with 8,192 tokens of context.