Babbage Index · Research · 008

The GPU Hour Is the Wrong Unit of Compute

A GPU hour measures resource occupancy, not productive computation. Priced as if it were, the cheaper hour can buy less useful work — so procurement, and any price index, needs a denominator that specifies what was delivered and under which constraints.
Babbage Research · 008·Published 2026-10-03

A GPU hour measures resource occupancy. It works for billing and capacity reservations. Treating it as a fixed amount of productive computation creates a pricing error: the cheaper hour can buy less useful work. An accelerator’s useful output depends on memory headroom, network topology, software, contention, and latency requirements. Procurement therefore needs a denominator that specifies what was delivered and under which constraints.

Consider two hypothetical serving configurations. One costs $2 per hour in compute rental and sustains 100 compliant requests per second; the other costs $3 and sustains 200. A request is compliant when it meets the quality threshold and its own latency deadline. Hold the model, quality threshold, prompt and output length distributions, context distribution, and latency targets constant, with sufficient demand to sustain both rates. Rental cost per million compliant requests is $5.56 versus $4.17. The second configuration costs 50% more per hour but 25% less per request. These are illustrative on-demand rental rates in US dollars for each whole configuration, not provider quotes. The calculation assumes sustained use at those rates and excludes networking, storage, and operations.

The crossover is explicit. Over the same observation window, the more expensive configuration wins when the ratio of its compliant work to the cheaper configuration’s exceeds the ratio of their total costs:

W2 / W1 > C2 / C1
W is compliant work completed and C is total cost, both over the same window; subscript 2 is the more expensive configuration.

Here, twice the work exceeds 1.5 times the rental cost. Adding the omitted costs can move that threshold or reverse the ranking. At lower demand, unused capacity can erase the throughput advantage. The threshold is exact: the $3 configuration stays cheaper per compliant request only while its utilization is above 75% of the $2 configuration’s. With demand of 100 requests per second for each, it costs $8.33 per million against $5.56, and the ranking reverses. Useful work also requires a workload-specific definition: training, interactive inference, and batch inference have different outputs and deadlines. A token count alone hides prompt versus generation work, context length, and quality.1,2

Keep resource units for reservations, and use workload-defined service units for acceptance. Specify the deployment configuration and a reproducible test covering quality, request mix, load, latency, and availability. For a price index, publish a versioned reference workload alongside the rate card.3 Changes in batching, numerical format, or request selection must remain visible because they can change the service being compared. Reproducibility does not guarantee representativeness. Differences between the reference workload and a buyer’s actual workload create basis risk: the index can move differently from the buyer’s effective cost. A workload-adjusted index should expose that gap.

Run a billing reconciliation against compliant goodput: useful work completed per second within the agreed quality and latency limits.1Connect the bill and total service cost to accepted work, then attribute lost output to unavailable capacity, queueing, retries, or deadline misses without double-counting. This diagnostic tests whether a low hourly price survives the conditions of production. If procurement or an index compares hours without a reproducible service contract, what exactly has become cheaper?

References

  1. Zhong, Liu, Chen, Hu et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” OSDI ’24. Goodput as the request rate served within time-to-first-token and time-per-output-token targets. arXiv:2401.09670
  2. Patel, Choukse, Zhang, Shah et al., “Splitwise: Efficient Generative LLM Inference Using Phase Splitting,” ISCA ’24. Prompt computation is compute-intensive and token generation memory-intensive, with distinct latency, throughput, memory and power characteristics. arXiv:2311.18677
  3. Reddi, Cheng, Kanter, Mattson et al., “MLPerf Inference Benchmark,” ISCA ’20. A rules-bound reference benchmark for comparing inference systems representatively and reproducibly. arXiv:1911.02549
Editor’s note

The question applies to the Babbage Index itself. BI-G is built from hourly per-GPU rate-card prices, so it measures the price of access — the reservation unit this note says the hour is good for — not the cost of completed work. BI-T is quoted per million tokens, a service unit, but its headline tracks output prices alone and so carries the prompt-versus-generation blind spot described above; input prices are published separately on the Token page. The hourly tier spreads in post 001 are differences in the price of access, not in the cost of delivered work. The two units also follow who controls the software: BI-G tracks bring-your-own-stack rental, where the buyer runs the software and the hour is the natural contract unit; BI-T tracks managed inference, where the provider runs it and per-token pricing shifts the performance risk onto the provider. Post 005 sets out the architecture terms that separate the two, and post 004 explains why the reference behind an index has to be reproducible by someone other than its publisher.

Provenance note, per house style. The worked example is illustrative and computed on this page from its stated inputs: 100 and 200 compliant requests per second are 0.36 and 0.72 million requests per hour, so $2 and $3 per hour come to $5.56 and $4.17 per million compliant requests. $3 is 50% above $2; $4.17 is 25% below $5.56. The break-even utilization is the cost ratio over the capacity ratio, 1.5 / 2 = 75%. The comparison excludes networking, storage, operations, and idle capacity, as the text states. No index data is used.