The Cheapest Node Can Waste the Scarcest Megawatt
Take an illustrative comparison. One hundred inference nodes that each need a 10 kW facility allocation, or 125 nodes that each need 8 kW and are each as productive as a 10 kW node, fill a 1 MW envelope. The 8 kW node costs 40% more. Post 008 settled what to count when comparing them — compliant goodput per dollar, not hours. This note asks the next question: once you count it, which constraint binds, dollars or watts? Post 005 already states the intuition in one line — a cheap GPU-hour is not cheap if it consumes scarce megawatts. What follows works out when that line holds and when it fails.
Two constraints, one model
The purchasing heuristic works while capital limits deployment: buy more compute and treat electricity as operating expense. A fixed electrical envelope adds a second constraint. For independently scaling replicas of one node type, achievable goodput is
Compliance here means the latency targets as well as quality. MLPerf’s Server and Interactive scenarios qualify LLM throughput the same way — for Llama 3.1 8B, time to first token of at most 2 s and time per output token of at most 100 ms in Server, 0.5 s and 30 ms in Interactive. Offline results carry no latency bound, so they are not g.3 When capital binds, the minimum reduces to ranking by g/c, which is 008’s crossover. The interesting case is the other one.
Power binding is necessary, not sufficient
With enough budget to fill the megawatt either way, the ranking collapses to g/w. The 8 kW fleet supplies 25% more goodput, but costs 1.75× as much to fill the megawatt. Three ratios describe the trade, and only one of them decides it:
| Measure | 10 kW fleet | 8 kW fleet |
|---|---|---|
| Goodput filling 1 MW (node-equivalents) | 100.0 | 125.0 |
| Capital to fill 1 MW (cost units) | 100.0 | 175.0 |
| Average capital per unit of goodput | 1.00 | 1.40 |
| Marginal capital per extra unit of goodput | — | 3.00 |
Both fleets draw the same megawatt, so energy cost cancels if utilization and service life match. The decision is therefore the margin on the extra 25 node-equivalents, which cost 75 extra cost units — 3.0 each. The premium pays only if each node-equivalent of goodput earns more than 3× the cheap node’s deployed cost over its life. Below that, the cheaper fleet makes more money from the same scarce megawatt even though it delivers less goodput. The general rule is to rank options by margin per binding watt — value of goodput, less annualized capital and energy, divided by w — with g/w as its limit when value far exceeds cost.
Between the regimes, buy both
The single-type model hides the most common case: a budget large enough that power binds for the cheap fleet but too small to fill the megawatt with the efficient one. In this example that is any budget between 100 and 175 cost units per megawatt. There, the best fleet is a mix that exhausts both constraints at once:
| Budget (cost units / MW) | All 10 kW | All 8 kW | Best mix |
|---|---|---|---|
| 100 | 100.0 | 71.4 | 100.0 |
| 120 | 100.0 | 85.7 | 106.7 |
| 140 | 100.0 | 100.0 | 113.3 |
| 160 | 100.0 | 114.3 | 120.0 |
| 175 | 100.0 | 125.0 | 125.0 |
| 200 | 100.0 | 125.0 | — |
The cheap node still wins outright below a budget of 140 even though power binds, and the pure options tie at 140. At that budget, 46.7 cheap nodes plus 66.7 efficient ones deliver 113.3 node-equivalents — 13% more than either pure choice. Only above 175 does the all-efficient fleet win on goodput, and then only subject to the margin test above. A scorecard that picks one SKU for the whole envelope misses the mix in every case between.
Oversubscription cuts both ways
Reclaimed headroom is real. Google reports power oversubscription of 25% or more across sharing domains of tens of megawatts, enabled by a new medium-voltage power plane and capping that coordinates workload priorities on every node, while preserving the availability and performance of all workloads.1 The same paper observes that spikes across rows and clusters are usually not correlated, which is why statistical sharing works. For LLM inference specifically, Patel et al. find that frequency locking reclaims up to 20% of peak power for at most 7% performance loss across the models they tested, and in simulation fit 30% more servers into the same cluster with minimal loss.2
None of this favors the cheap node by itself, because the same policy applies to the efficient fleet. The fair comparison oversubscribes both. If only the cheap fleet is oversubscribed by 25% to match the efficient fleet’s node count, it does so at 1.25× rather than 1.75× the capital — and then wins if each capped node keeps more than 87% of its goodput (at a value of 3× cost per node-equivalent). That retention is not measured anywhere public: capping a 10 kW node to 8 kW is a 20% node-level cut, beyond the characterized range. It has to be benchmarked on the actual workload and GPU generation. The 8 kW node is favored when both fleets get the same oversubscription, when the required cut exceeds what has been characterized, for larger models and larger prompt batches — where Patel et al. measure the biggest losses — and for latency-strict traffic, which has the least slack to absorb a throttle. The 10 kW node is favored where light caps suffice.
Capped capacity also fails non-linearly. In an M/M/1 queue at 900 requests per second, cutting service capacity from 1000 to 920 — an 8% throttle — raises mean response time from 10 ms to 50 ms, and SLOs are written on the tail, not the mean (see 007).
The diagnostic: a power-response curve
The quantity every branch above depends on is how much compliant goodput a workload gains or loses per kilowatt. Published methods characterize that curve;2 few operators calibrate it per workload and hardware generation. Join time-aligned facility power, cap residency, queue depth, throttling state and completed-request telemetry, then calibrate with controlled allocation changes. Utilization and average watts cannot answer the counterfactual, and neither can fleet-average PUE: it is a period-averaged ratio of facility to IT energy across a site, while the binding limit is coincident peak power in one domain. Without the curve, procurement-induced capacity loss can be misread as scheduler inefficiency, and a site can end up ordering machines it cannot energize.
References
- Sakalkar, Kontorinis, Landhuis, Li, De Ronde, Blooming, Ramesh, Kennedy, Malone, Clidaras, Ranganathan, “Data Center Power Oversubscription with a Medium Voltage Power Plane and Priority-Aware Capping,” ASPLOS 2020. research.google
- Patel, Choukse, Zhang, Goiri, Warrier, Mahalingam, Bianchini, “Characterizing Power Management Opportunities for LLMs in the Cloud,” ASPLOS ’24, doi:10.1145/3620666.3651329; preprint as “POLCA: Power Oversubscription in LLM Cloud Providers,” arXiv:2308.12908. One study, two versions.
- Iyengar, Atta-Fosu, Kong, “MLPerf Inference 5.1: Benchmarking Small LLMs with Llama3.1-8B,” MLCommons, September 9, 2025. mlcommons.org
Provenance note, per house style. Illustrative: every wattage, cost and value in the example (10 kW, 8 kW, a 40% premium, a 1 MW envelope, value of 3× cost) and the M/M/1 rates. Computed on this page from those inputs: the goodput and capital ratios, the marginal cost of 3.0, the budget table and mixed fleet, the 86.7% retention threshold (one minus the 0.4 cost premium divided by the value of 3), and the response times. Quoted: the Google oversubscription result and its observation on uncorrelated spikes (ref 1); the 20%/7% frequency-locking range and the simulated 30% server gain (ref 2); the MLPerf latency limits (ref 3).