Babbage Index · Research · 015

A Checkpoint Is Only as Useful as Its Restore Path

Checkpoint engineering mostly optimizes the write. But as asynchronous saving drives the write cost toward zero, the restore — reading the copy back, into replacement capacity, often alongside other jobs doing the same — becomes most of what a failure costs, and every second of it is billed.
Babbage Research · 015·Published 2026-10-24

Much checkpoint engineering targets the write side. Check-N-Run attacks the write bandwidth and capacity that bottleneck checkpoint frequency;1 Llama 3’s training infrastructure aimed “to minimize GPU pause time during checkpointing and increase checkpoint frequency to reduce the amount of lost work after a recovery.”2 That focus made sense when failures were rare. At scale they are not: the same report counts 466 job interruptions in a 54-day snapshot, 47 of them planned.2 Recovery-side systems exist too — GEMINI places in-memory checkpoints so that recovery from CPU memory is likely to survive a failure,3 and ByteCheckpoint reshards at load time4 — but even recovery-oriented metrics tend to stop at one job’s retrieval. GEMINI’s own accounting excludes “the overheads to fix failures and replace machines.”3

A written checkpoint is not a bounded recovery time. The path back may need different bandwidth, more memory, replacement workers, metadata reconstruction, and a topology that no longer exists after the failure that made the checkpoint necessary.

The restart term is in the theory

The standard periodic model already carries it. With checkpoint interval T, exposed save cost c, job failure rate λ and restore time r, the fraction of time lost is approximately5,6,7

c / T + λ · ( T / 2 + r )
r runs from failure to resumed useful work: detection, capacity acquisition, the restore read, state rebuild, validation and warmup. λ is the job’s rate — roughly the per-node rate times the node count. The model assumes independent, rare failures and uniformly distributed lost progress.

If r is constant, reducing it raises goodput without changing the optimal T; Daly’s higher-order model reaches the same conclusion — “R has no contribution” to the optimum.6 Restore time sets the size of the loss, not the interval. And the loss is where the inversion shows. At the optimum T* = √(2c/λ), the loss is √(2cλ) + λr. As asynchronous saving drives the exposed cost c toward zero, the first term vanishes and λr is what remains:

Exposed save cost cOptimal interval T*Save + reworkRestore (r = 320 s)
10 s465 s4.30%2.96%
1 s147 s1.36%2.96%
0.1 s46 s0.43%2.96%

That is at an illustrative one failure every three hours. Restore overtakes the save-and-rework term once r exceeds T*. Practice and reporting drop the term the theory keeps: a fast save figure is quoted; the route back is not.

Restores share the path

Take an illustrative 4 TB checkpoint whose shared restore path sustains 100 GB/s of read bandwidth. Reading it alone takes at least 40 s. If 8 jobs, each restoring its own 4 TB checkpoint, start at once and share that path equally, each read takes 320 s — before capacity acquisition or state rebuild. Admitting them one at a time keeps the 320 s makespan but cuts the mean to 180 s, and the idle job-seconds from 2,560 to 1,440. The periodic model cannot price this case: a correlated incident breaks its independence assumption, and with K concurrent restores r becomes r(K). The 320 s is the read component of r conditional on that incident, not an input the formula covers.

Placement compounds it. The copies easiest to write may sit in the same failure domain as the failure, while the durable remote copy needs the slow path exactly when many jobs want it. In-memory and peer-replicated copies cut r only for failures their domain survives. A fast asynchronous save shrinks c; it says nothing about this contention, or about the lag between snapshot and durable completion.

Every second of restore is billed

During a restore the whole reserved job is billed and does no work. Idle cost per incident is roughly the number of accelerators times r times the hourly rate, and the expected idle share of the bill is simply λr — 2.96% of reserved spend at the illustrative rate above, for the read component alone. This is the price of access held idle, which is exactly what an hourly price measures (008); restore time sits inside the goodput accounting 005 describes. Storage, meanwhile, is usually bought by capacity, while a restore consumes bandwidth — the line where 001 found the storage premium sitting.

A recovery contract

Specify recovery before choosing checkpoint placement: which failures a copy survives, how freshness is established, what replacement topology is acceptable, and which resources a restore may reserve. Separate durable completion from background transfer still in flight. Include model state, optimizer state where needed, learning-rate schedule and step counters, data-loader position, random state and coordination metadata — CheckFreq checkpoints data-loader state precisely so each item is still used once per epoch after a resume.8 Set an explicit priority between restores and background saves, because one storage network cannot promise both unrestricted bandwidth, and admit concurrent restores deliberately rather than letting every job saturate the path at once.

Then test it: restore from the surviving copy into replacement capacity under realistic concurrent recovery, not by loading a local file into idle machines, and measure time to resumed useful work, including validation and warmup. A checkpoint is valuable because it limits lost work and restores service, not because it produced a high write bandwidth figure. The reliability budget that stops counting when the write returns is pricing the cheap half of the route.

References

  1. Eisenman et al., “Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models,” NSDI ’22. usenix.org
  2. Llama Team, AI @ Meta, “The Llama 3 Herd of Models,” §3.3. arXiv:2407.21783
  3. Wang et al., “GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints,” SOSP ’23.
  4. Wan et al., “ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development.” arXiv:2407.20143
  5. Young, “A First Order Approximation to the Optimum Checkpoint Interval,” Communications of the ACM 17(9), 1974.
  6. Daly, “A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps,” Future Generation Computer Systems 22(3), 2006.
  7. Benoit, Du, Herault, Marchal, Pallez, Perotin, Robert, Sun, Vivien, “Checkpointing à la Young/Daly: An Overview,” IC3 2022.
  8. Mohan, Phanishayee, Chidambaram, “CheckFreq: Frequent, Fine-Grained DNN Checkpointing,” FAST ’21. usenix.org
Editor’s note
No Babbage index prices recovery. BI-G is an hourly access price and BI-S a capacity price; neither carries restore bandwidth or the idle hours a restore consumes, which is why the cost here is stated as a share of the reserved bill rather than in dollars.

Provenance note, per house style. Illustrative: the 4 TB checkpoint, the 100 GB/s path, the 8 concurrent jobs, the one-failure-per-three-hours rate and the save costs. Computed on this page from them: the 40 s and 320 s reads, the 180 s staggered mean and idle job-seconds, the optimal intervals and the loss shares. Quoted: the Llama 3 checkpointing aim and interruption counts (ref 2), GEMINI’s exclusion of machine replacement (ref 3), Daly’s finding that the restart term does not move the optimum (ref 6), and CheckFreq’s data-loader checkpointing (ref 8).