ModelVRAM
EN 中文

What LLMs can an L40S run?

An L40S (48 GB) runs 40 of the 69 open models tracked here at Q4_K_M with 8K context; the largest is Llama 3.1 70B (70.6B parameters, 47.0 GB). It reads memory at 864 GB/s. 3 more fit at a lower precision. Each fit leaves at least 0.5 GB free on the L40S (why a card does not give a model all its memory).

Not sure your PC has an L40S? Detect your GPU in the browser.

Models that fit on one L40S

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first. gpt-oss-20b (14.8 GB, 87–154 tokens/s on the L40S) is listed at MXFP4 only, as published.

Model Best precision Memory Longest context Tokens/s
AliceAI Foundation 80B-A3B 81.3B, 3B active GGUF Q3_K_M 41.4 GB 244K 126–229
Qwen3-Coder-Next 79.7B, 3B active GGUF Q3_K_M 40.6 GB 256K (full) 126–229
Kolibri-1 78.1B, 3.5B active GGUF Q3_K_M 39.9 GB 256K (full) 110–198
Llama 3.1 70B 70.6B GGUF Q4_K_M 47.0 GB 9K 10–14
K2-Horizon MoVA 36B-A4B 37.4B, 4B active GGUF Q8_0 42.9 GB 30K 41–71
Ornith 1.5 35B-A3B 36.0B, 3B active GGUF Q8_0 39.8 GB 256K (full) 69–121
Qwen3.6 35B-A3B 36.0B, 3B active GGUF Q8_0 39.8 GB 256K (full) 69–121
Ornith 1.0 35B 35.1B, 3B active GGUF Q8_0 38.9 GB 256K (full) 69–121
LLM-jp-4.1 32B-A3B Thinking 32.1B, 3.8B active GGUF Q8_0 36.0 GB 64K (full) 52–90
Nemotron 3 Nano 30B-A3B 31.6B, 3.5B active GGUF Q8_0 34.9 GB 256K (full) 62–108
Gemma 4 31B 31.3B GGUF Q8_0 36.5 GB 135K 13–18
GLM-4.7 Flash 31.2B, 3B active GGUF Q8_0 34.9 GB 198K (full) 64–112
Xing 4.0 29B-A4B 31.2B, 4B active GGUF Q8_0 34.9 GB 256K (full) 52–89
Qwen3-Coder 30B-A3B 30.5B, 3.3B active GGUF Q8_0 34.6 GB 133K 55–95
Qwen3 30B-A3B 30.5B, 3.3B active GGUF Q8_0 34.6 GB 40K (full) 55–95
Muse Glimmer 30B 29.8B GGUF Q8_0 33.1 GB 128K (full) 15–20
Granite 4.2 30B 29.3B GGUF Q8_0 34.6 GB 54K 14–19
ThinkingCap Qwen3.8 27B 27.8B GGUF Q8_0 31.3 GB 243K 15–21
Qwen3.6 27B 27.8B GGUF Q8_0 31.3 GB 243K 15–21
Qwen3.8 27B 27.8B GGUF Q8_0 31.3 GB 243K 15–21
Hemmingway-1 27B 27.3B GGUF Q8_0 30.8 GB 251K 16–22
Gemma 4 26B-A4B 25.8B, 4B active GGUF Q8_0 29.1 GB 256K (full) 51–87
gpt-oss-20b 20.9B, 3.6B active As published (MXFP4) 14.8 GB 128K (full) 87–154
Gemma 4 12B 12.0B FP16 / BF16 25.7 GB 256K (full) 19–26
ZDTaichu 5.0 9B 9.8B FP16 / BF16 20.8 GB 128K (full) 23–32
Ornith 1.5 9B 9.7B FP16 / BF16 20.6 GB 256K (full) 23–33
Qwen3.5 9B 9.7B FP16 / BF16 20.6 GB 256K (full) 23–33
LensVLM 9B 9.4B FP16 / BF16 20.1 GB 256K (full) 24–33
Ornith 1.0 9B 9.4B FP16 / BF16 20.1 GB 256K (full) 24–33
MiMo V2.6 Distill Qwen 9B 9.4B FP16 / BF16 20.1 GB 256K (full) 24–33
Bespoke Nimble 9B 9.0B FP16 / BF16 19.1 GB 8K (full) 25–35
Granite 4.2 8B 8.8B FP16 / BF16 19.9 GB 128K (full) 24–34
LFM2.5 8B-A1B 8.5B, 1.5B active FP16 / BF16 18.0 GB 128,000 (full) 74–130
Qwen3 8B 8.2B FP16 / BF16 18.5 GB 40K (full) 26–36
Llama 3.1 8B 8.0B FP16 / BF16 18.1 GB 128K (full) 27–37
Gemma 4 E4B 8.0B FP16 / BF16 17.1 GB 128K (full) 28–39
Ling 3.0 Tiny 7.9B-A1.3B 7.9B, 1.3B active FP16 / BF16 16.7 GB 128K (full) 85–150
LightOnOCR-3 4B 4.5B FP16 / BF16 10.1 GB 256K (full) 47–67
Spark-X2.5 4B 4.1B FP16 / BF16 9.35 GB 994K 51–72
Nemotron 3 Nano 4B 4.0B FP16 / BF16 8.78 GB 256K (full) 54–77
Granite 4.2 3B 3.7B FP16 / BF16 8.69 GB 128K (full) 55–78
MiniCPM5 2B 2.5B FP16 / BF16 6.02 GB 128K (full) 78–113
Limite 1B Violetto 1.0B FP16 / BF16 2.79 GB 128K (full) 161–253

Too big for one L40S at 8K context

How many L40S cards these need at Q4_K_M (gpt-oss-120b at MXFP4) with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.

Best models for L40S with room for long context

The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the card takes at that precision and the writing speed with that context in the cache, from 864 GB/s of bandwidth.

ModelContextMemoryLeft overLongest contextTokens/s
K2-Horizon MoVA 36B-A4B 37.4B 32K 30.3 GB 17.7 GB 115K 28–48
Ornith 1.5 35B-A3B 36.0B 32K 23.5 GB 24.5 GB 256K (full) 90–160
Qwen3.6 35B-A3B 36.0B 32K 23.5 GB 24.5 GB 256K (full) 90–160
Ornith 1.0 35B 35.1B 32K 22.9 GB 25.1 GB 256K (full) 90–160
LLM-jp-4.1 32B-A3B Thinking 32.1B 32K 22.6 GB 25.4 GB 64K (full) 53–92

llama-server commands for one L40S

GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on the L40S. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.

K2-Horizon MoVA 36B-A4B, Q4_K_M with 115K tokens: 10.0–17 tokens/s on the L40S

llama-server -hf IFM/K2-Horizon-MoVA-36B-A4B-GGUF:Q4_K_M -c 117760

K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf, 22.4 GB: 47.4 GB used and 587 MB free of the L40S's 48 GB, 10.0–17 tokens/s once the 115K cache is full.

Ornith 1.5 35B-A3B, Q4_K_M with 256K tokens: 34–58 tokens/s on the L40S

llama-server -hf bartowski/Ornith-1.5-35B-A3B-GGUF:Q4_K_M -c 262144

Ornith-1.5-35B-A3B-Q4_K_M.gguf, 21.9 GB: 28.4 GB used and 19.6 GB free of the L40S's 48 GB, 34–58 tokens/s once the 256K cache is full.

How fast the L40S writes as the context fills

Tokens per second at Q4_K_M for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the card holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 864 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside. With the same memory, M4 Pro Mac (64 GB), 273 GB/s, writes 68% slower by bandwidth; 2× RTX 3090, 1,872 GB/s, writes 117% faster by bandwidth; 2× RTX 4090, 2,016 GB/s, writes 133% faster by bandwidth.

Model8K32KLongestAt the longestQ8_0 at 8KBandwidth ceiling at 8K1,000-token reply at 32K
K2-Horizon MoVA 36B-A4B 59–102 28–48 115K 10.0–17 41–71 214 21–36 s
Ornith 1.5 35B-A3B 109–196 90–160 256K 34–58 69–121 436 6–11 s
Qwen3.6 35B-A3B 109–196 90–160 256K 34–58 69–121 436 6–11 s
Ornith 1.0 35B 109–196 90–160 256K 34–58 69–121 436 6–11 s
LLM-jp-4.1 32B-A3B Thinking 80–141 53–92 64K 37–63 52–90 303 11–19 s
Nemotron 3 Nano 30B-A3B 101–181 96–170 256K 63–110 62–108 399 6–10 s
Gemma 4 31B 22–31 20–28 256K 11–15 13–18 41 36–50 s
GLM-4.7 Flash 98–175 65–114 198K 20–33 64–112 383 9–15 s
Xing 4.0 29B-A4B 81–143 60–104 256K 17–29 52–89 309 10–17 s
Qwen3-Coder 30B-A3B 81–143 46–80 256K 9.2–15 55–95 308 13–22 s

Near misses on L40S

Models that miss at Q4_K_M with 32,768 tokens of context (or their whole window), or leave under 0.5 GB free there, but fit with a shorter context or 3-bit weights, or are short by at most a quarter of the memory and fit with MoE offload. Smallest shortfall first.

ModelShort by at Q4_K_MFits instead
Kolibri-1 78.1B 1.70 GB GGUF Q3_K_M with 32K, 40.4 GB
Qwen3-Coder-Next 79.7B 2.71 GB GGUF Q3_K_M with 32K, 41.2 GB
AliceAI Foundation 80B-A3B 81.3B 3.71 GB GGUF Q3_K_M with 32K, 42.0 GB
Llama 3.1 70B 70.6B 7.23 GB Q4_K_M with 9K context

What more cards or MoE offload unlock

One, two and four L40S cards in one tensor-parallel group (vLLM or SGLang), with the models each step adds at Q4_K_M and 8K context and their tokens per second on that group, and the median on-demand rent for that many cards (getdeploying.com, 2026-09-29).

Setup Memory Bandwidth Models at Q4_K_M Rent per hour Adds (tokens/s)
L40S 48 GB 864 GB/s 40 $1.57 This page
2× L40S 96 GB 1,728 GB/s 48 $3.14 Kolibri-1 124–248, Qwen3-Coder-Next 135–275, AliceAI Foundation 80B-A3B 135–275, gpt-oss-120b 104–201, Nemotron 3 Super 120B-A12B 57–102, Nemotron 3.5 Super VL 120B-A12B 57–102, Qwen3.5 122B-A10B 64–117, Mistral Medium 3.5 128B 11–16
4× L40S 192 GB 3,456 GB/s 53 $6.28 Qwen3.8 Flash Next 139–284, Step 3.7 Flash 196B-A11B 96–183, MiniMax M2.7 88–167, DeepSeek V4 Flash 85–160, DeepSeek V4 Flash 0731 85–160

Every mixture-of-experts GGUF file the --n-cpu-moe planner has measured, on one L40S with 32K tokens of context (or the model's whole window): the smallest --n-cpu-moe that fits the card (0 means the whole file fits), the memory on the card and in system RAM, and tokens per second from 864 GB/s of card bandwidth with DDR5-5600, 2 channels (89.6 GB/s) or DDR4-3200, 2 channels (51.2 GB/s). 3 of these models do not fit whole and need offload.

Model GGUF --n-cpu-moe On the card In system RAM Tokens/s, DDR5 Tokens/s, DDR4
gpt-oss-20b MXFP4 0 12.5 GB 587 MB 68–120 68–120
Gemma 4 26B-A4B UD-Q4_K_M 0 17.0 GB 748 MB 64–111 64–111
Qwen3 30B-A3B Q4_K_M 0 21.1 GB 167 MB 47–81 47–81
Qwen3 30B-A3B Q8_0 0 33.9 GB 315 MB 38–64 38–64
GLM-4.7 Flash UD-Q4_K_XL 0 18.8 GB 170 MB 57–98 57–98
GLM-4.7 Flash Q4_K_M 0 19.5 GB 170 MB 57–98 57–98
Nemotron 3 Nano 30B-A3B Q4_K_M 0 23.8 GB 231 MB 82–145 82–145
Qwen3.6 35B-A3B UD-IQ4_XS 0 17.6 GB 515 MB 73–128 73–128
Ornith 1.5 35B-A3B Q4_K_M 0 21.6 GB 273 MB 84–149 84–149
Qwen3.6 35B-A3B UD-Q4_K_M 0 21.7 GB 515 MB 70–123 70–123
Qwen3.6 35B-A3B Q8_0 0 35.5 GB 515 MB 61–106 61–106
Ornith 1.5 35B-A3B Q8_0 0 36.3 GB 515 MB 60–104 60–104
Qwen3-Coder-Next IQ4_XS 0 41.3 GB 167 MB 76–133 76–133
Qwen3-Coder-Next Q4_K_M 0 46.8 GB 167 MB 71–124 71–124
Qwen3-Coder-Next UD-Q4_K_XL 0 47.6 GB 315 MB 58–101 58–101
gpt-oss-120b MXFP4 8 48.0 GB 13.2 GB 29–50 22–37
Qwen3.5 122B-A10B Q4_K_M 18 47.8 GB 25.2 GB 16–27 12–20
Qwen3.8 Flash Next UD-Q2_K_XL 0 48.0 GB 417 MB 46–79 46–79
Qwen3.8 Flash Next UD-IQ3_XXS 3 47.8 GB 3.48 GB 39–66 36–62
Qwen3.8 Flash Next GSQ-RCO IQ3_S 5 47.9 GB 4.89 GB 36–62 33–56
Qwen3.8 Flash Next UD-Q3_K_XL 9 47.7 GB 11.0 GB 29–50 25–42
Qwen3.8 Flash Next UD-IQ4_XS 12 47.3 GB 14.9 GB 27–46 22–37
Qwen3.8 Flash Next UD-Q4_K_XL 20 48.0 GB 30.6 GB 20–34 15–25
Qwen3.8 Flash Next UD-Q5_K_XL 27 46.2 GB 52.3 GB 15–25 10–17

In system RAM counts the offloaded experts plus the input embedding (token_embd), which llama.cpp keeps in RAM in any case.

Run locally or rent an L40S?

Renting an L40S costs a median of $1.57 an hour on demand, across the providers getdeploying.com tracks (checked ), so every $1,000 of purchase price equals about 637 hours of rent, 27 days around the clock. At that rate K2-Horizon MoVA 36B-A4B at Q4_K_M with 32K tokens, the biggest model here that fits with a long context, writes 28–48 tokens/s for one request: about $9.2–16 per million generated tokens. Electricity and resale value are left out, and no card prices are listed because 2026 street prices vary widely between sellers; compare current ones before you decide.

Serving with vLLM on the L40S

The KV cache pool vLLM builds at its default --gpu-memory-utilization 0.92 and --max-model-len 32768, weights as published, from the vLLM KV cache calculator. The pool is shared by every request; the last column assumes 4,096 tokens each.

ModelKV cache per GPUPool (tokens)At 32KRequests at 4K
gpt-oss-20b 25.8 GB 996,626 30.41x 137
Qwen3 8B 24.1 GB 175,424 5.35x 42

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Nearby GPUs and setups

How many of the tracked models each one holds at Q4_K_M with 8K context, against 40 here.

SetupHow it relatesMemoryBandwidthModels at Q4_K_M
M4 Pro Mac (64 GB) Same memory 48 GB usable 273 GB/s 40 (same)
2× RTX 3090 Same memory 48 GB 1,872 GB/s 40 (same)
2× RTX 4090 Same memory 48 GB 2,016 GB/s 40 (same)
RTX 5090 Next size down 32 GB 1,792 GB/s 39 (−1)
2× RTX 5090 Next size up 64 GB 3,584 GB/s 43 (+3)

Model numbers read from Hugging Face on .