FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.
Quantization is usually sold as a free lunch: same quality, less memory, faster decoding. We wanted to check that claim before recommending FP8 anywhere, so we benchmarked it the way it actually ships — the real service, under real concurrency — instead of a single offline pass. Setup We served Qwen2.5-7B-Instruct on an H100 with vLLM 0.31.0, comparing bf16 against online FP8, a calibrated FP8 checkpoint, and AWQ across a concurrency sweep. Every comparison went through the same gates we use for engine upgrades: latency split into TTFT vs. TPOT, output-quality checks with significance testing, and cost converted to dollars per 1M tokens. What we found FP8 did what it promises on latency: TPOT dropped 23–26%. −23 to −26% time per output token, FP8 vs. bf16 97.6% → 75% self-consistency at 64 concurrent requests, bf16 vs. FP8 65 / 70 FP8 outputs that changed with batch size, vs. 35 / 70 for bf16 What it also did was less comfortable. At 64 concurrent requests,
Quantization is usually sold as a free lunch: same quality, less memory, faster decoding. We wanted to check that claim before recommending FP8 anywhere, so we benchmarked it the way it actually ships — the real service, under real concurrency — instead of a single offline pass.
Setup
We served Qwen2.5-7B-Instruct on an H100 with vLLM 0.31.0, comparing bf16 against online FP8, a calibrated FP8 checkpoint, and AWQ across a concurrency sweep. Every comparison went through the same gates we use for engine upgrades: latency split into TTFT vs. TPOT, output-quality checks with significance testing, and cost converted to dollars per 1M tokens.
What we found
FP8 did what it promises on latency: TPOT dropped 23–26%.
−23 to −26%
time per output token, FP8 vs. bf16
97.6% → 75%
self-consistency at 64 concurrent requests, bf16 vs. FP8
65 / 70
FP8 outputs that changed with batch size, vs. 35 / 70 for bf16
What it also did was less comfortable. At 64 concurrent requests, self-consistency — how often the service gives the same answer to the same prompt — fell from 97.6% under bf16 to about 75% under FP8.
To find out why, we ran the same prompts at batch sizes from 1 to 128. Rerunning an input at the same batch size gave identical output on both precisions, so the engine itself is deterministic. Changing the batch size is what changed the text: FP8's output changed in 65 of 70 cases, with the first differing token at a median of 19. bf16 changed too, in 35 of 70 cases, but much later, at a median of token 77. Some batch dependence is normal; FP8 nearly doubled it and moved it much earlier in the answer.
| BATCH-SIZE PROBE | SAME INPUT, RERUN | DIFFERENT BATCH SIZE | MEDIAN FIRST DIFFERING TOKEN |
|---|---|---|---|
| bf16 | identical | 35 / 70 changed | 77 |
| FP8 | identical | 65 / 70 changed | 19 |
In other words, FP8 doesn't just make decoding faster. Under load, its answers depend on how busy the server is: cheaper per token, but measurably less consistent with itself.
Why this matters
If you only watch latency, FP8 looks like an obvious win. If you watch latency, quality, and cost together, it's a tradeoff with a real price tag: lower cost per 1M tokens, paid for in output stability. That's exactly the kind of tradeoff that should be decided with data, not with a benchmark table from a launch post.
What this doesn't prove
These numbers are from one model, our prompts, and our serving setup. Quantization behavior varies by model family, and some of our capacity numbers were measured only once. Where the tokens still matched, FP8's next-token distribution barely moved, and its quality drop on our prompt set was too small to be statistically significant. We publish the methodology and the limitations alongside the results because a benchmark you can't reproduce is marketing, not evidence.
What we'd do instead
Gate quantization changes the same way you'd gate an engine upgrade: run the real service against a committed baseline in CI, require statistical significance before calling anything a regression (noise gets marked inconclusive, not regressed), and check quality at the concurrency you actually serve — not at batch size 1.
See it on your own model
Browse the verdicts from these runs in the read-only demo, no account needed. Or run a pilot: we measure your next quantization or engine change on your own service and prompts.
Explore the live demo
Request a pilot
.jpg)
