What LLMs can an A100 80GB run?

Models that fit on one A100 80GB

The most precise weights that still fit with 8K tokens of context and one request, the memory that takes, the longest context at that precision and the writing speed. Bigger models come first. gpt-oss-120b (67.7 GB, 149–276 tokens/s on the A100 80GB) and gpt-oss-20b (14.8 GB, 174–329 tokens/s on the A100 80GB) are listed at MXFP4 only, as published.

Too big for one A100 80GB at 8K context

How many A100 80GB cards these need at Q4_K_M with 8K context and 0.5 GB to spare, in one tensor-parallel group; some need more than 8.

• Step 3.7 Flash 196B-A11B: 2 cards
• MiniMax M2.7: 2 cards
• DeepSeek V4 Flash: 4 cards
• DeepSeek V4 Flash 0731: 4 cards
• MiMo V2.6 Flash: 4 cards
• IQuest-Q1 320B-A15B: 4 cards
• GLM-5.3 Flash: 4 cards
• MiniMax M3: 4 cards
• Ling 3.1 Flash 560B-A25B: 8 cards
• DeepSeek V3 / R1: 8 cards
• DeepSeek V3.2: 8 cards
• GLM-5.3: 8 cards
• GLM-5.2: 8 cards
• DeepSeek V4.1 Flash 1017: 8 cards
• DeepSeek V4.1 Flash: 8 cards
• Hy4 Preview 770B-A49B: 8 cards
• MiMo V2.6 Pro: 8 cards
• DeepSeek V4 Pro: more than 8 cards
• Qwen3.8 2.4T-A95B: more than 8 cards
• Kimi K3: more than 8 cards

Best models for A100 80GB with room for long context

The biggest models that still fit at Q4_K_M with 32,768 tokens of context (or their whole window, if shorter), the memory left over, the longest context the card takes at that precision and the writing speed with that context in the cache, from 2,039 GB/s of bandwidth. gpt-oss-120b is at MXFP4, as published.

llama-server commands for one A100 80GB

GGUF repos checked 2026-09-29; -c is the longest context with 0.5 GB free on the A100 80GB. No -ngl: llama.cpp’s -fit, on by default since b7440, places the layers.

Qwen3.5 122B-A10B, Q4_K_M with 22K tokens: 81–143 tokens/s on the A100 80GB

llama-server -hf unsloth/Qwen3.5-122B-A10B-GGUF:Q4_K_M -c 22528

Q4_K_M/Qwen3.5-122B-A10B-Q4_K_M-00001-of-00003.gguf, 76.5 GB: 79.5 GB used and 537 MB free of the A100 80GB's 80 GB, 81–143 tokens/s once the 22K cache is full.

gpt-oss-120b, MXFP4 with 128K tokens: 71–124 tokens/s on the A100 80GB

llama-server -hf ggml-org/gpt-oss-120b-GGUF:MXFP4 -c 131072 -np 1

gpt-oss-120b-MXFP4.gguf, 63.4 GB: 72.3 GB used and 7.68 GB free of the A100 80GB's 80 GB, 71–124 tokens/s once the 128K cache is full. -np 1: one slot, one sliding window.

How fast the A100 80GB writes as the context fills

Tokens per second at Q4_K_M (gpt-oss-120b at MXFP4) for the biggest models that fit with 32K tokens, with 8K and 32K tokens in the cache and at the longest context the card holds, and at Q8_0 with 8K where that fits. Each token reads the active weights and the whole cache once, so the speed follows the 2,039 GB/s of bandwidth and falls as the cache grows. The ceiling is that bandwidth divided by the bytes read per token, which no runtime reaches; the reply time is for 1,000 new tokens with 32K (or the longest context) in the cache, prompt processing aside. With the same memory, H100 SXM, 3,350 GB/s, writes 49% faster on average.

A100 80GB against the other 80 GB cards

The same models fit on every card with 80 GB, so speed is what separates them. By bandwidth the A100 80GB is the slowest of the 2 cards with 80 GB at 2,039 GB/s. Over the 12 biggest models that fit at Q4_K_M with 8K context, the H100 SXM (3,350 GB/s) writes 49% faster than the A100 80GB. Each cell below is the other card's tokens per second minus this one's, midpoints of the estimated ranges.

Near misses on A100 80GB

Models that miss at Q4_K_M with 32,768 tokens of context (or their whole window), or leave under 0.5 GB free there, but fit with a shorter context or 3-bit weights, or are short by at most a quarter of the memory and fit with MoE offload. Smallest shortfall first.

What more cards or MoE offload unlock

One, two and four A100 80GB cards in one tensor-parallel group (vLLM or SGLang), with the models each step adds at Q4_K_M and 8K context and their tokens per second on that group, and the median on-demand rent for that many cards (getdeploying.com, 2026-09-29).

Run locally or rent an A100 80GB?

Renting an A100 80GB costs a median of $1.76 an hour on demand, across the providers getdeploying.com tracks (checked 2026-09-29), so every $1,000 of purchase price equals about 568 hours of rent, 24 days around the clock. At that rate Qwen3.5 122B-A10B at Q4_K_M with 32K tokens, the biggest model here that fits with a long context, writes 79–138 tokens/s for one request: about $3.5–6.2 per million generated tokens. Electricity and resale value are left out, and no card prices are listed because 2026 street prices vary widely between sellers; compare current ones before you decide.

Serving with vLLM on the A100 80GB

The KV cache pool vLLM builds at its default --gpu-memory-utilization 0.92 and --max-model-len 32768, weights as published, from the vLLM KV cache calculator. The pool is shared by every request; the last column assumes 4,096 tokens each.

How these numbers are worked out

Memory is the weights at each precision plus the KV cache for 8,192 tokens in FP16 and runtime overhead (0.5 GB plus 10%), computed from each model's files on Hugging Face with the LLM VRAM Calculator. Speed is estimated from memory bandwidth and the parameters read per token with the LLM Speed Calculator. Both tools take any other model from Hugging Face.

Nearby GPUs and setups

How many of the tracked models each one holds at Q4_K_M with 8K context, against 47 here.