Babbage Index · Research · 017

One CPU SKU Is Not One Latency Distribution

Same-SKU processors differ electrically, and under a power or thermal limit that difference can become a difference in service rate. Near saturation, a queue turns a small service-rate gap into a large latency gap — so a fan-out request can miss its deadline because one host in a hundred is a few percent slow.
Babbage Research · 017·Published 2026-10-31

006 and 007 showed that nominally identical GPUs differ, and that a synchronous cohort pays for its slowest member. This note carries the argument to CPUs serving fan-out requests, where the amplifier is different: not the maximum over a cohort, but queueing near saturation. A request that needs every shard misses its deadline if any shard is late, and how late a shard runs depends on its slack — service rate minus arrival rate — times the time remaining.

Electrical spread is measured; capacity spread is conditional

In a June 2026 preprint, Crop, Moore and Pasricha measured ten Xeon 6314 (Ice Lake) parts of the same SKU at 2.0 GHz and an average die temperature of 50°C, and found core voltage varying by up to 40 mV, leakage differing by nearly 2×, and dynamic capacitance by up to 13%.1 That establishes electrical heterogeneity. Measured at a fixed frequency, it is a difference in power, not throughput.

It becomes a capacity difference mainly when a power or thermal limit binds. Inadomi et al. show the conversion directly.2 Without a cap, 2,386 Sandy Bridge processors showed almost no performance variation despite up to 23% variation in power. Under RAPL caps on a 1,920-module Ivy Bridge system, cutting the module budget for DGEMM from 110 W to 70 W widened the CPU frequency spread from 1.20× to 1.40×, and module power variation produced up to 64% performance variation across application ranks. The conversion is not automatic: for MHD, a 60 W cap produced a 1.75× frequency spread while per-rank execution time stayed within about 1%, because, as the authors put it, CPU-boundedness, memory and synchronization characteristics decide the impact. And “mainly” is not “only”: one uncapped system of 64 processors showed 17% performance variation alongside 21% in power. All of this is 2013-era HPC under deliberate caps, not stock-limit cloud servers — evidence for the mechanism, not a calibration of how slow a production host runs.

The queueing amplifier

Model each shard as an independent M/M/1 queue — Poisson arrivals, exponential service, first-come first-served, a single-server abstraction of a many-core host. A fan-out request that must wait for all N shards misses its remaining deadline D with probability

1 − Πi [ 1 − e−(μi − λi)D ]
λi < μi are each shard’s arrival and service rates. Keeping the miss rate within ε needs roughly (μi − λi)D ≥ ln(N/ε) on every shard — here 9.2.

Take 100 shards at 900 req/s each and 100 ms of remaining deadline. With every shard at 1000 req/s, the request miss rate is 0.45%. Now make one shard run at 920 req/s — an 8% shortfall chosen purely for illustration, not measured on any part and unrelated to the 8% GPU statistic 006 discussed. The miss rate jumps to 13.9%. Whatever the dependence between shards, if that host’s own queue behaves as M/M/1, it alone imposes a 13.5% floor. Against a 1% budget, any single-shard shortfall above about 4.8% breaches it at this load — and the load matters as much as the shortfall:

Load per shardAll healthyOne shard at 920 req/s
850 req/s0.00%0.09%
880 req/s0.06%1.89%
900 req/s0.45%13.92%
910 req/s1.23%37.56%

Headroom or placement

There are two ways to buy the budget back. Cut load fleet-wide to about 874 req/s — 2.9% of capacity — and the miss rate returns to 1%. Or give the weak host only what it can carry: sizing its shard to 92% of the load restores 0.46% at a cost of 0.08% of aggregate capacity. Headroom looks affordable here only because the worst host is known. Across a fleet nobody has measured, headroom has to be sized for a deficit nobody has observed; placement needs the measurement, but then costs almost nothing.

None of the tools is new. Dean and Barroso prescribed micro-partitions and latency-induced probation for slow machines,3 and PAL places GPU work by measured, per-application variability.4 Hedged requests and retries remain right for transient stragglers; repeatedly hedging a persistently weak host becomes an operating subsidy. What is missing is upstream: CPU procurement acceptance that measures service curves across supported power limits and hands schedulers capability classes before a host takes traffic. The qualification sweep itself is the one 011 describes; the measured curve is 008’s workload-defined service unit at host scale.

The diagnostic

The question to ask of each host is counterfactual: how would the same request stream perform at matched temperature, power limit, NUMA placement and software state? Join queue wait and service time with effective frequency (APERF/MPERF counters), package energy, throttle reasons where the platform exposes them, memory traffic and workload identity, and use controlled replay and component swaps to separate persistent silicon effects from cooling, firmware and interference. A tenant who cannot see the hardware has a cruder version of the same tool: benchmark at launch, and replace hosts that come up slow.

References

  1. Crop, Moore, Pasricha, “Revisiting ‘Cooler is Better’: ITD-Aware Per-CPU Thermal Optimization for Sustainable Data Center Operation,” preprint, §3. arXiv:2606.11163
  2. Inadomi et al., “Analyzing and Mitigating the Impact of Manufacturing Variability in Power-Constrained Supercomputing,” SC ’15. doi:10.1145/2807591.2807638
  3. Dean, Barroso, “The Tail at Scale,” Communications of the ACM 56(2), 2013. research.google
  4. Jain, Tran, Chen, Sinclair, Venkataraman, “PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU Clusters,” SC ’24. arXiv:2408.11919
Editor’s note
BI-C prices access: cents per vCPU-hour, the same for every host of a SKU. It says nothing about the service rate a given host delivers, so it cannot show the variation this note is about — and two buyers paying the same BI-C rate can get different capacity.

Provenance note, per house style. Quoted: the Xeon 6314 voltage, leakage and capacitance spreads (ref 1); the uncapped and capped power, frequency and performance variation figures (ref 2). Illustrative: 100 shards, 900 and 1000 req/s, the 920 req/s weak shard, 100 ms of deadline and the 1% budget. Computed on this page from them: every miss rate, the 13.5% single-shard floor, the 4.8% breach threshold, the slack rule, the sensitivity table, and both remedies.