Per-Token Prices Miss the Cost of Long Context
A language model generates text one token at a time. Each step, every attention layer reads the keys and values it stored for all earlier tokens — the KV cache — so the bytes read per step grow with the token’s position in the context. Most price lists do not. Anthropic’s page says that for Claude 4.6 and later models (except Haiku 5.5) “a 900k-token request is billed at the same per-token rate as a 9k-token request.”3 The serving mechanism is sound. The unit of account on top of it measures tokens, while the resource consumed is bytes kept resident and re-read.
The conclusion is not new to Babbage Research. 005 lists “$/token” and “$/resident-token” as different goods and says decode buys memory bandwidth and resident-token capacity; 018 notes that when KV bandwidth binds, “residence time is the demand”; and 008 argues that a token count hides context length. This article supplies the arithmetic behind those one-line claims: how fast the cost grows, where it overtakes the fixed cost of reading the model, what real price cards charge for it, and what would have to be metered to price it.
One convention throughout: token counts are decimal, as on price cards (K = 1,000 and M = 1,000,000), and so are bytes (GB = 10⁹, TB = 10¹²). Prices are in US dollars per million tokens (MTok), standard tier, as listed on 2026-11-08.
What one step reads
Take Llama 3 70B, whose shape is public: 80 layers, 8 KV heads (grouped-query attention shares each among 8 query heads), and a head dimension of 128.1 At two bytes per element (bf16), each token leaves a key and a value vector in every layer:
That is 327.68 kB per token of context. A decode step at position 8,000 reads 2.62 GB of KV for that one sequence, which takes 0.33 ms at a B200’s peak 8 TB/s of HBM bandwidth.2 At position 1,000,000 the step reads 327.7 GB, 41.0 ms. Both times are lower bounds: real kernels achieve less than peak bandwidth. The million-token case is also 1.8× one B200’s 180 GB of HBM, so it must be spread across devices; tensor parallelism divides the wall-clock time, not the bytes. Both of those output tokens carry the same price.
Where context starts to dominate
Each step also reads the model’s weights, but that read is shared by every sequence in the batch. Counting the same table’s matrices gives 70.55 billion parameters, or 141.1 GB at bf16. With an illustrative 64 concurrent sequences, each sequence’s share of the weight read is 2.20 GB per step. The KV read equals that share at:
Below about 6.7K tokens, the shared weight read, which really is the same for every token, is most of each step’s traffic, so a flat price is off by less than 2×. But Claude 4.6 and later models ship with a 1M-token context window,3 and Gemini 3.1 Pro prices prompts above 200K.4 Above the crossover, a sequence’s traffic per step grows in proportion to its position:
| Context position (tokens) | KV read per step (GB) | Weight share per step (GB) | KV share of traffic | KV read time at peak (ms) |
|---|---|---|---|---|
| 2,000 | 0.66 | 2.20 | 23% | 0.08 |
| 6,728 | 2.20 | 2.20 | 50% | 0.28 |
| 32,000 | 10.49 | 2.20 | 83% | 1.31 |
| 200,000 | 65.54 | 2.20 | 97% | 8.19 |
| 1,000,000 | 327.68 | 2.20 | 99% | 40.96 |
The table holds for bandwidth-bound decode of this model at batch 64. S* scales inversely with batch: at 32 sequences it is about 13,456, at 128 about 3,364. The batch is the operator’s choice, which is part of the problem — the same token costs different amounts depending on who it is batched with.
The batch is also capped by memory. At batch 64, the KV for every sequence fits in an eight-GPU B200 node’s 1.44 TB of HBM, alongside one copy of the weights, only below about 61,936 tokens.2 Beyond that, capacity sets the batch: an operator can hold only about (HBM − W)/(k·S) sequences, so each sequence’s share of a step grows in proportion to S. That reaches the same conclusion by a different route. The table’s rows above that line illustrate the per-step read, not a feasible batch.
Two requests, one bill structure
Compare a request with a 200,000-token prompt and a 4,000-token completion against the same completion after a 2,000-token prompt. Decode step j reads k·(S + j) bytes, so the whole completion reads:
The long request’s decode reads 264.8 TB of KV, about 33.1 B200-seconds at peak bandwidth; the short one reads 5.24 TB. That is 50.5× the KV traffic. Adding each request’s share of the weight read (8.8 TB over 4,000 steps at batch 64), total decode traffic differs by 19.5×. The bills, at three USD price cards:
| Price card (input / output, $ per MTok) | 2,000 in, 4,000 out | 200,000 in, 4,000 out | Bill ratio |
|---|---|---|---|
| Claude Sonnet 4.6 (Anthropic): $3 / $15 | $0.066 | $0.660 | 10.0× |
| Claude Opus 5.5 (Anthropic): $4 / $20 | $0.088 | $0.880 | 10.0× |
| Gemini 3.1 Pro Preview, ≤200K tier (Google): $2 / $12 | $0.052 | $0.448 | 8.6× |
At the two Anthropic cards, whose input-to-output ratio is 1:5, the long request bills 10.0× the short one; at Gemini 3.1 Pro’s rates (a 200,000-token prompt still falls in its ≤200K tier) it bills 8.6×. The bill ratio depends only on the card’s input-to-output ratio, not its level, which is why 10.0× repeats for Opus 5.5. In every case the extra charge comes from the input term, which is linear in prompt length and pays for prefill — processing the prompt once. (Prefill attention is not linear: at a 200,000-token prompt on this shape, causal attention takes about 1.9× the FLOPs of the 2·P·S matrix work, a separate cross term the input price also misses.) Nothing on the invoice scales with the 4,000 times the prompt is re-read during decode. Because $0.448 and $0.660 are list prices, not costs, the comparison says nothing about any provider’s margin; it says the bill’s shape does not follow the traffic’s. Token counts are held fixed across cards; Opus 5.5’s newer tokenizer produces about 30% more tokens for the same text,3 so the dollar levels are not a like-for-like comparison of the same document.
What the existing fixes price
Providers do charge for context in other ways, but each charge prices something adjacent to residency.
Tiers. Gemini 3.1 Pro Preview charges $2.00 input and $12.00 output per MTok for prompts up to 200K tokens, and $4.00 and $18.00 above.4 The tier is set by the prompt’s size, so a step function sits on a linear cost. A 200,001-token prompt pays 2.0× the input bill of a 200,000-token one for one extra token. A 1,000,000-token prompt pays 1.5× the output rate above the threshold, while each of its decode steps reads 5× the KV of a step at the threshold (4.9× including the weight share). Per output token, the tier overcharges just above 200K and undercharges far above it. Per request the 2.0× input step can dominate instead: a 1,000,000-token prompt with a 4,000-token completion bills 9.1× a 200,000-token one for about 5.0× the decode KV traffic. Anthropic does the same for Claude Haiku 5.5, whose prompts over 100,000 tokens pay $0.50/$2.50 instead of $0.10/$0.50 per MTok, a 5.0× step.3 Either way, a step tariff sits on a cost that grows smoothly with position.
Cache reads. A cache hit reuses a stored prompt prefix instead of recomputing it. Anthropic prices hits at 0.1× base input for most models, 0.05× for Claude Opus 5.5 and Sonnet 5.5, and 0.025× for Claude Fable 5.2, Mythos 5.2, Fable 5.1 and Mythos 5.1;3 Gemini 3.1 Pro’s $0.20 (prompts up to 200K) is 0.1× its $2.00 input rate.4 These discounts price avoided prefill — the trade 010 works through — not the decode re-read, which happens whether the prefix came from cache or not.
Storage. Gemini charges $4.50 per million tokens per hour to keep a context cached;4 holding a 200,000-token prefix for an hour costs $0.90. That prices residency, but only between requests, while nothing is being decoded against it. The live decode, when every resident byte is read every step, carries no residency charge at all.
Admission. Inside a serving engine, the knob that limits how much context is resident is a count. vLLM’s max_num_seqs caps concurrent sequences, and when the paged KV pool (the block allocator from PagedAttention5) runs out, the engine preempts requests and recomputes them later; its tuning guide suggests lowering max_num_seqs to reduce preemptions.6 A count does not distinguish a 2K sequence from a 200K one, so a few long contexts slow every sequence they share a batch with, and the preemptions show up as a tail in time per output token. Each of these mechanisms lives in one price card or one replica; none sees aggregate bandwidth.
Making it cheaper changes the slope, not the shape
Storing the KV cache in FP8 halves k and doubles the crossover to about 13,456 tokens. Multi-head latent attention goes further by caching a compressed latent instead of full keys and values: DeepSeek-V3 caches a 512-element latent plus a 64-element positional key in each of 61 layers,7 which at two bytes per element is 70,272 bytes per token, 4.7× less than Llama 3 70B’s. That is a different model with different weights, so its crossover is not directly comparable, but the shape is the same: per-step KV traffic is still k times position, with a smaller k. Better kernels and formats move the crossover out; they do not make a position-blind price track cost above it.
Where the argument holds and where it fails
The arithmetic assumes decode is limited by memory bandwidth and that every resident KV byte is read every step. That holds for standard full-attention models at the batch sizes interactive services run. It weakens or fails when:
- the batch is large and the context short enough that matrix arithmetic, not memory traffic, sets the step time;
- the model uses sliding-window or local attention in some layers, which caps the bytes those layers read regardless of position;
- many sequences share a prefix and the kernel reads it once per batch rather than once per sequence;
- the engine evicts or compresses old KV entries, trading accuracy for bandwidth; or
- the workload is mostly prompt and little output, so prefill dominates; the input price is linear in prompt length while prefill attention grows with its square, so the position error moves to the input side rather than disappearing.
Closed models do not publish their shapes, so the Llama figures illustrate the mechanism; they are not an estimate of any provider’s cost.
Meter what is consumed
Under the bandwidth model, a sequence’s traffic at position S is W/B + k·S = (W/B)(1 + S/S*). A cost-reflective output price would scale by that factor: 30.7× at position 200,000, relative to a token at position 0, for this illustrative shape and batch. Averaged over the two completions above it is 1.59× and 31.0×, the same 19.5× gap as the total traffic. No provider needs to publish that curve to fix the accounting. The quantity to meter is bytes resident times steps alive, which equals KV bytes read, because every resident byte is read every step. Summed per tenant it becomes a KV byte-read budget (bytes resident × decode steps) that a cluster scheduler, which sees every replica, can use for admission, placement and tariff. Multiplying by step time instead gives byte-seconds, which also charges idle residency — the quantity Gemini’s storage fee prices between requests.
The documentation we checked does not expose that quantity. DCGM’s memory-bandwidth metric (DRAM_ACTIVE, the fraction of cycles with device-memory traffic) is device-level, and the guide lists no DRAM byte counter;8 vLLM’s KV metrics report block usage and block lifetimes, labeled by model, and its per-request metrics count tokens and time, not bytes.9 An engine already knows each sequence’s length, so it could compute k·S per step from its block tables; a per-sequence byte counter in the attention kernel would verify it. Without either, a few long-context tenants look like a fleet-wide latency regression and a capacity shortfall, and the remedy bought is more GPUs instead of a different tariff.
What this means for compute pricing and Babbage’s indices
A per-token price is not one quantity of compute. The same output token can carry under a millisecond or tens of milliseconds of HBM bandwidth depending on where it sits. A token-price benchmark that does not state a context-length distribution is averaging goods of very different cost — the same specification gap 014 identifies for every compute service, from GPU-hours to storage: specify the service before pricing it. To compare token prices across providers or over time, a benchmark would need to fix prompt and output lengths, cache-hit share, KV precision where disclosed, and a latency or concurrency class.
BI-T sees list prices per million tokens at the base tier. Babbage’s committed token history on 2026-11-08 records Claude Sonnet 4.6 at $3 / $15, Claude Opus 5.5 at $4 / $20, and Gemini 3.1 Pro at $2 / $12 — the ≤200K tier — with no field for the long-context tier, cache prices or storage fees. Its headline tracks US-frontier output prices alone, as 008 notes. So BI-T would not move if a provider raised its long-context tier or cache-storage fee, and it would not move if the mix of traffic shifted toward long contexts and raised the cost of serving the average token. BI-G, which prices GPU-hours, sees the capacity that long contexts consume — HBM and its bandwidth, rented by the hour — but not how that capacity is divided among tokens. Neither index sees the cross term. Adding context tiers to BI-T would be a methodology change requiring review; this article proposes none and records the gap.
References
- Grattafiori et al., “The Llama 3 Herd of Models,” arXiv:2407.21783, Table 3. arxiv.org
- NVIDIA, HGX AI Factory reference architecture, components, Table 1 (B200: 180 GB HBM3e, up to 8 TB/s). docs.nvidia.com
- Anthropic, Claude API pricing: model pricing, prompt caching multipliers, long context pricing (read 2026-11-08). platform.claude.com
- Google, Gemini Developer API pricing, Gemini 3.1 Pro Preview, standard tier (read 2026-11-08). aistudio.google.com
- Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP ’23. arxiv.org
- vLLM, Optimization and tuning: preemption and
max_num_seqs. docs.vllm.ai - DeepSeek-AI, “DeepSeek-V3 Technical Report,” arXiv:2412.19437, §2.1.1 and model hyperparameters. arxiv.org
- NVIDIA, DCGM feature overview, profiling metrics (DCGM_FI_PROF_DRAM_ACTIVE). docs.nvidia.com
- vLLM, Metrics design. docs.vllm.ai
Provenance note, per house style. Quoted: Llama 3 70B’s layer, head, width and vocabulary figures (ref 1); B200 HBM capacity and peak bandwidth (ref 2); Anthropic’s list prices, cache multipliers and long-context sentence (ref 3); Gemini 3.1 Pro Preview’s tiers, cache-read and storage prices (ref 4); Haiku 5.5’s prompt-length tiers and the tokenizer note (ref 3); DeepSeek-V3’s latent dimensions (ref 7); the engine and counter behavior (refs 6, 8, 9). Computed on this page: k, the parameter count, S*, per-step reads and times, request traffic, bills and ratios, and the position multipliers. Assumed or illustrative: bf16 weights and KV, batch 64, an eight-GPU node, peak bandwidth, the prefill FLOP model, the 200,000/2,000/4,000-token requests, bf16 for DeepSeek-V3’s cache, and the bandwidth-bound decode model.