What LLMs can eight H100 SXM GPUs run?

Models that fit on 8× H100 SXM

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first. Memory is for the whole group: weights, cache and 10% split across the 8 cards, plus 0.5 GB of runtime on each card (4 GB in all). gpt-oss-120b (71.2 GB, 257–651 tokens/s on 8× H100 SXM) and gpt-oss-20b (18.3 GB, 262–670 tokens/s on 8× H100 SXM) are listed at MXFP4 only, as published.

Too big for 8× H100 SXM at 8K context

How many H100 SXM cards these need at Q4_K_M with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.

• DeepSeek V4 Pro: more than 8 cards
• Qwen3.8 2.4T-A95B: more than 8 cards
• Kimi K3: more than 8 cards

Best models for 8× H100 SXM with room for long context

The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the group takes at that precision and the writing speed with that context in the cache, from 26,800 GB/s of bandwidth. Memory is for the whole group: weights, cache and 10% split across the 8 cards, plus 0.5 GB of runtime on each card (4 GB in all). Memory and left over add up to the group's 640 GB.

llama-server commands for 8× H100 SXM

GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on 8× H100 SXM. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.

GLM-5.2, Q4_K_M with 1M tokens: 54–97 tokens/s on 8× H100 SXM

llama-server -hf unsloth/GLM-5.2-GGUF:Q4_K_M -c 1048576

UD-Q4_K_M/GLM-5.2-UD-Q4_K_M-00001-of-00011.gguf, 465.8 GB: 581 GB used and 59.4 GB free of 8× H100 SXM's 640 GB, 54–97 tokens/s once the 1M cache is full.

How fast 8× H100 SXM writes as the context fills

Tokens per second at Q4_K_M for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the group holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 26,800 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside.

Near misses on 8× H100 SXM

Models that miss at Q4_K_M with 32,768 tokens of context (or their whole window), or leave under 0.5 GB free there, but fit with a shorter context or 3-bit weights, or are short by at most a quarter of the memory and fit with MoE offload. Smallest shortfall first.

Run locally or rent eight H100 SXM GPUs?

Renting eight H100 SXM GPUs costs a median of $27.12 an hour on demand ($3.39 per card), across the providers getdeploying.com tracks, whose H100 figure covers every H100 variant (checked 2026-09-29), so every $1,000 of purchase price equals about 37 hours of rent, 2 days around the clock. At that rate Hy4 Preview 770B-A49B at Q4_K_M with 32K tokens, the biggest model here that fits with a long context, writes 132–267 tokens/s for one request: about $28–57 per million generated tokens. Electricity and resale value are left out, and no card prices are listed because 2026 street prices vary widely between sellers; compare current ones before you decide.

Serving with vLLM on 8× H100 SXM

The KV cache pool vLLM builds with --tensor-parallel-size 8 at its default --gpu-memory-utilization 0.92 and --max-model-len 32768, weights as published, from the vLLM KV cache calculator. The pool is shared by every request; the last column assumes 4,096 tokens each.

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB on each card plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Nearby GPUs and setups

How many of the tracked models each one holds at Q4_K_M with 8K context, against 66 here.