Babbage Index · Research · 018

Speculative Decoding Can Spend More Than It Saves

Speculative decoding buys lower latency with extra computation. Whether it also lowers the cost of a token depends on which resource binds — and that changes with load and context length. A single-stream speedup quoted without its load says nothing about fleet capacity.
Babbage Research · 018·Published 2026-11-04

Speculative decoding trades additional computation for fewer sequential target-model steps. A small draft proposes tokens; the target verifies them in one pass and keeps those its sampling rule accepts. The papers that introduced it scoped the trade honestly: it is “not helpful for configurations where additional computation resources are not available.”1 Serving work since has measured the consequence — under higher request rates or low speculation accuracy, speculation “paradoxically increases latency.”2 What persists is a reporting habit: single-stream speedups published without the load at which they were measured. This note is about the units that make the trade legible, and what it means for a price per token.

Demand per emitted token

Let d and v be the draft and verification demand a round places on one constrained resource, n̄ the long-run mean number of tokens emitted per round — including the target’s bonus or correction token, so 1 ≤ n̄ ≤ γ + 1 — and b the baseline demand per output token on the same resource, at the same batch size and load. Speculation reduces demand on that resource when

E[ d + v ] / E[ n̄ ] < b
A ratio of long-run means, not a mean of per-round ratios. It is Leviathan et al.’s total-operations result (Theorem 3.11) restated as demand on one bottleneck.1

Consider illustrative values, not measurements: γ = 4, b = 10 units, a deliberately expensive draft at d = 8 (the original paper’s drafts cost under 5% of the target1), and verification at v = 12 — about one target step, as it is when decode is memory-bound at low batch. At n̄ = 3 the round costs 6.7 units per token against 10. The break-even is n̄* = 2, which at γ = 4 corresponds to a per-token acceptance rate of about 0.52. A cheap draft lowers the bar: d = 2 gives n̄* = 1.4, and as d → 0 it tends to 1.2.

Load changes the verification cost

That example shows acceptance risk. Load is the larger risk, because it changes v. At low batch, decode is bound by reading weights and KV state, and verifying γ + 1 positions costs roughly one step. At high batch with short contexts, decode becomes arithmetic-bound, and verifying γ + 1 positions costs roughly γ + 1 steps of arithmetic:

Tokens emitted per round n̄Memory-bound (v = 12)Arithmetic-bound (v = 50)Baseline b
120.058.010.0
210.029.010.0
36.719.310.0
54.011.610.0

In the arithmetic-bound column no acceptance rate helps: because n̄ never exceeds γ + 1, demand per token is at least b + d/n̄, always above b. Speculation never reduces arithmetic per token; it saves bytes — weight and KV reads per emitted token. That is 005’s split in miniature: prefill buys arithmetic, decode buys bandwidth, and speculation spends the first to save the second.

Occupancy can reverse the conclusion again. When the binding resource is KV bandwidth or cache capacity — moderate-to-long contexts — residence time is the demand, and shortening it can raise capacity even at large batch. MagicDec reports up to 2.51× speedup for Llama-3.1-8B at batch sizes from 32 to 256 on moderate-to-long sequences,3 net of the draft’s own footprint. “Disable speculation near saturation” is right for short-context, arithmetic-bound serving and wrong for long-context, bandwidth-bound serving.

Choosing depth across requests

The batch scheduler, together with admission, should therefore choose speculation depth — including zero — by the marginal tokens emitted per unit of the binding resource. That controller exists: SmartSpec, later TurboSpec, sets speculation length per step from a goodput estimate,2 and vLLM exposes a switch that disables speculation above a configured batch size.4 The coordination matters because no stream can assume idle target cycles will remain once every stream starts drafting. As a design judgment rather than a measured result: switching modes has a cost, because re-enabling speculation forces the draft to rebuild its cache state, so a controller should not flip on noisy acceptance estimates.

The comparison is also only fair if the output is the same service. Speculative sampling with modified rejection sampling preserves the target distribution “within hardware numerics.”5 Lossy variants exist — Medusa’s typical acceptance states plainly that it does “not insist on an exact correspondence between the output and language model distribution”6 — and those buy a different product, which has to be compared as one.

Latency is a product; capacity is a cost

Where speculation is visible in pricing at all, the buyer pays for the spend. OpenAI’s Predicted Outputs — user-supplied predictions, not a model-drafted speculation, but the same accept-or-reject mechanics — note that “any rejected tokens are still billed like other completion tokens generated by the API, so Predicted Outputs can introduce higher costs for your requests.”7 A technique that improves interactive latency can be worth buying even when it spends more compute. The mistake is reporting a latency gain as a capacity gain: the first is a product, the second a cost line, and 008’s compliant-goodput accounting is where they meet.

References

  1. Leviathan, Kalman, Matias, “Fast Inference from Transformers via Speculative Decoding,” ICML 2023, Theorem 3.11 and Conclusion. arXiv:2211.17192
  2. Liu et al., “Optimizing Speculative Decoding for Serving Large Language Models Using Goodput” (SmartSpec, v1, June 2024), revised as “TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput.” arXiv:2406.14066
  3. Sadhukhan, Chen, Chen et al., “MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding.” arXiv:2408.11049
  4. vLLM v0.6.3 engine arguments, speculative_disable_by_batch_size. github.com/vllm-project
  5. Chen, Borgeaud, Irving, Lespiau, Sifre, Jumper, “Accelerating Large Language Model Decoding with Speculative Sampling.” arXiv:2302.01318
  6. Cai et al., “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.” arXiv:2401.10774
  7. OpenAI, “Predicted Outputs” guide. platform.openai.com
Editor’s note
BI-T quotes prices per million tokens with no latency class. A provider that serves the same model faster by spending more compute per token has the same BI-T entry as one that does not, so the index cannot show the capacity cost of a latency product — the gap between $/token and $/useful-token that 005 names.

Provenance note, per house style. Illustrative: γ = 4, b = 10, d = 8, and both verification costs. Computed on this page from them: every per-token demand, the break-even n̄* = 2 and its acceptance rate of 0.52 (from Leviathan’s expected-tokens formula), and the table. Quoted: Leviathan’s scope statement and draft cost (ref 1), SmartSpec’s finding (ref 2), MagicDec’s 2.51× (ref 3), the vLLM switch (ref 4), the exactness and lossy-acceptance statements (refs 5–6), and OpenAI’s billing note (ref 7).