ToolSite LLM Speed Calculator
Pick a model and a GPU, choose the quantization and context length, and see how fast it writes: the memory-bandwidth limit and the range that engines such as llama.cpp and vLLM reach for one request.
Advanced: active parameters and several GPUs
– tokens per second
- Bandwidth limit
- –
- Read per token
- –
- Of which KV cache
- –
- A 500-token answer
- –
On other hardware
| GPU | Bandwidth | Tokens per second |
|---|---|---|
| RTX 4060 8GB | 272 GB/s | – |
| RTX 3060 12GB | 360 GB/s | – |
| Arc B580 12GB | 456 GB/s | – |
| RTX 4070 12GB | 504 GB/s | – |
| RTX 5070 12GB | 672 GB/s | – |
| RTX 4060 Ti 16GB | 288 GB/s | – |
| RTX 5060 Ti 16GB | 448 GB/s | – |
| RX 9070 XT 16GB | 644 GB/s | – |
| RTX 4070 Ti Super 16GB | 672 GB/s | – |
| RTX 4080 Super 16GB | 736 GB/s | – |
| RTX 5070 Ti 16GB | 896 GB/s | – |
| RTX 5080 16GB | 960 GB/s | – |
| RTX 3090 | 936 GB/s | – |
| RTX 4090 | 1,008 GB/s | – |
| RX 7900 XTX | 960 GB/s | – |
| RTX 5090 | 1,792 GB/s | – |
| M4 Pro Mac (64 GB) about 75% of unified memory is usable by the GPU | 273 GB/s | – |
| M4 Max Mac (128 GB) about 75% of unified memory is usable by the GPU | 546 GB/s | – |
| M3 Ultra Mac Studio (512 GB) about 75% of unified memory is usable by the GPU | 819 GB/s | – |
| Ryzen AI Max+ 395 (128 GB) up to 96 GB can be assigned to the GPU | 256 GB/s | – |
| DGX Spark (128 GB) | 273 GB/s | – |
| L40S | 864 GB/s | – |
| RTX PRO 6000 Blackwell | 1,792 GB/s | – |
| A100 80GB | 2,039 GB/s | – |
| H100 SXM | 3,350 GB/s | – |
| H200 | 4,800 GB/s | – |
| B200 | 8,000 GB/s | – |
Speed of popular models
Tokens per second for one request at Q4_K_M with a 4,096-token context; a dash means the model does not fit on one card. The model name leads to its memory requirements.
| Model | RTX 3060 12GB | RTX 4090 | RTX 5090 | M4 Max Mac (128 GB) | H100 SXM |
|---|---|---|---|---|---|
| Qwen3.6 35B-A3B (MoE) | — | 129–234 | 199–382 | 76–134 | 295–612 |
| Qwen3.6 27B | — | 31–43 | 53–76 | 17–24 | 93–137 |
| Qwen3-Coder-Next (80B MoE) | — | — | — | 76–133 | 294–608 |
| Gemma 4 31B | — | 27–37 | 46–65 | 15–20 | 81–119 |
| Gemma 4 26B-A4B (MoE) | — | 97–172 | 155–287 | 56–97 | 240–477 |
| Gemma 4 E4B | 38–53 | 96–143 | 154–240 | 56–80 | 240–406 |
| GLM-4.7 Flash | — | 121–220 | 189–361 | 72–126 | 284–583 |
| Nemotron 3 Nano 30B-A3B (MoE) | — | 116–210 | 182–346 | 69–120 | 275–562 |
| gpt-oss-20b | — | 111–199 | 174–328 | 65–113 | 265–537 |
| gpt-oss-120b | — | — | — | 47–81 | 212–411 |
| Qwen3.5 9B | 32–44 | 81–119 | 132–202 | 47–66 | 211–347 |
| Llama 3.1 8B | 35–49 | 89–131 | 143–222 | 51–73 | 226–378 |
How to use
- Choose a model, or type a Hugging Face id such as Qwen/Qwen3.6-35B-A3B and press Load.
- Pick your GPU or Mac and the weight precision you plan to run.
- Set the context length: the longer the conversation, the more cache every token has to read.
- Read the speed range and the table of other GPUs. Open Advanced to change the active parameters or run on several GPUs.
FAQ
How is the speed estimated?
Writing one token means reading the active weights and the whole KV cache from GPU memory once, so the ceiling is the memory bandwidth divided by those bytes. Llama 3.1 8B at Q4_K_M reads about 4.9 billion bytes per token, so an RTX 4090 with 1,008 GB/s tops out near 205 tokens per second.
What speed do real engines reach?
For one request, llama.cpp and vLLM typically reach 55–75% of that ceiling with dense models and 30–50% with mixture-of-experts models, whose expert routing costs more, and every token also costs about 0.5–1.5 ms for launching the GPU kernels, which caps small models on fast GPUs. Llama 3.1 8B at Q4_K_M runs at about 130 tokens per second on an RTX 4090.
Why are MoE models faster than their size suggests?
Each token reads only the experts it is routed to. Qwen3-30B-A3B reads about 3.3 billion of its 30.5 billion parameters per token, so it writes faster than an 8B dense model even though it needs more memory.
Why does it slow down in long conversations?
The whole KV cache is read for every new token. At 128K tokens the FP16 cache of Llama 3.1 8B is 16 GB, more than three times its Q4_K_M weights, so it writes about four and a half times slower than at the start of a conversation.
Do two GPUs make it twice as fast?
Not quite. With tensor parallelism, as in vLLM or SGLang, each card reads half of every layer, so the bandwidth limit doubles, but the cards exchange results after every layer, which the calculator counts as about 1–2 ms per token. llama.cpp splits whole layers across cards by default, which adds memory but writes at about the speed of one card.
Does reading the prompt follow the same rule?
No. Processing the prompt (prefill) is limited by compute rather than bandwidth and runs at hundreds to thousands of tokens per second on a modern GPU. This calculator covers the writing speed, which is what you notice in a chat.
What about Macs and other unified-memory machines?
The same rule applies with their memory bandwidth: 546 GB/s for an M4 Max, 819 GB/s for an M3 Ultra and 256 GB/s for a Ryzen AI Max+ 395. They hold bigger models than most GPUs but write more slowly.
Is anything I enter sent anywhere?
The calculation runs in your browser. The only network requests go to Hugging Face, when you load a model.
More tools
Updated