LLM Speed Calculator

Advanced: active parameters and several GPUs

Parameters read per token (billions)

GPUs in tensor parallel 1248

Speed of popular models

Tokens per second for one request at Q4_K_M with a 4,096-token context; a dash means the model does not fit on one card. The model name leads to its memory requirements.

Time to first token with a 32K-token prompt

Q4_K_M, one request, central estimate; “—” where the model does not fit the card or the card has no FP16 figure.

How the prompt time is estimated

Reading a prompt is limited by compute, not memory bandwidth: every prompt token runs through every active weight once, 2 floating-point operations per weight, and attention adds 4 × width per query-key pair per layer, which grows with the square of the prompt length (sliding-window layers only up to their window; linear-attention and Mamba layers not at all). Time = operations ÷ (share × the card’s dense FP16 tensor throughput from the maker’s spec sheet), and attention runs at 0.9× the share of the weight matrices. The share is fitted per family on the public llama-bench runs below: 0.59 for GeForce and RTX, 0.19 for A100/H100/H200, which llama.cpp does not drive near their peak, and 0.4 for Radeon with ROCm. Mixture-of-experts models count only their active parameters, at 0.51× that share, fitted on gpt-oss 20B. One share for all cards would be off by -20% to +270%; per family, every run below lands within ±28%.

Calibration: estimate vs public llama-bench runs

Llama 2 7B Q4_0 from llama.cpp’s CUDA and ROCm scoreboards (flash attention on, one GPU), plus Llama 3.1 8B and gpt-oss 20B from the benchmark table of llama.cpp PR #25319. ppN is llama-bench’s average speed reading an N-token prompt into an empty cache, so N ÷ ppN is the wait for the first token.

FP16 tensor throughput used

Dense FP16 tensor throughput with FP16 accumulate where NVIDIA lists both; AI TOPS are divided by the ratio each whitepaper shows for cards of the same architecture. Cards not listed here get no prompt estimate.

Measured prompt speed on Macs

Apple publishes no FP16 tensor throughput, so these are measurements rather than estimates: Llama 2 7B Q4_0, pp512, from llama.cpp’s Apple Silicon thread (the chip’s largest GPU configuration). For a model with P billion active parameters, divide by about P ÷ 6.7; longer prompts are slower still. The M5 generation reads prompts 3.3–3.7 times faster than M4 with the same number of GPU cores.

Run locally or rent a GPU?

Renting an RTX 4090 costs a median of $0.53 an hour on demand and an H100 $3.39, across the providers getdeploying.com tracks (checked 2026-09-29; its H100 figure covers every H100 variant). The table shows how many hours of rent every $1,000 of card price equals, before electricity and resale value, and what Qwen3.6 35B-A3B at Q4_K_M with 4,096 tokens costs per million generated tokens for one request. No card prices are listed because 2026 street prices vary widely between sellers; compare current ones before you decide. Serving several requests at once on a rented card lowers the cost per token.