A Checkpoint Is Only as Useful as Its Restore Path
Much checkpoint engineering targets the write side. Check-N-Run attacks the write bandwidth and capacity that bottleneck checkpoint frequency;1 Llama 3’s training infrastructure aimed “to minimize GPU pause time during checkpointing and increase checkpoint frequency to reduce the amount of lost work after a recovery.”2 That focus made sense when failures were rare. At scale they are not: the same report counts 466 job interruptions in a 54-day snapshot, 47 of them planned.2 Recovery-side systems exist too — GEMINI places in-memory checkpoints so that recovery from CPU memory is likely to survive a failure,3 and ByteCheckpoint reshards at load time4 — but even recovery-oriented metrics tend to stop at one job’s retrieval. GEMINI’s own accounting excludes “the overheads to fix failures and replace machines.”3
A written checkpoint is not a bounded recovery time. The path back may need different bandwidth, more memory, replacement workers, metadata reconstruction, and a topology that no longer exists after the failure that made the checkpoint necessary.
The restart term is in the theory
The standard periodic model already carries it. With checkpoint interval T, exposed save cost c, job failure rate λ and restore time r, the fraction of time lost is approximately5,6,7
If r is constant, reducing it raises goodput without changing the optimal T; Daly’s higher-order model reaches the same conclusion — “R has no contribution” to the optimum.6 Restore time sets the size of the loss, not the interval. And the loss is where the inversion shows. At the optimum T* = √(2c/λ), the loss is √(2cλ) + λr. As asynchronous saving drives the exposed cost c toward zero, the first term vanishes and λr is what remains:
| Exposed save cost c | Optimal interval T* | Save + rework | Restore (r = 320 s) |
|---|---|---|---|
| 10 s | 465 s | 4.30% | 2.96% |
| 1 s | 147 s | 1.36% | 2.96% |
| 0.1 s | 46 s | 0.43% | 2.96% |
That is at an illustrative one failure every three hours. Restore overtakes the save-and-rework term once r exceeds T*. Practice and reporting drop the term the theory keeps: a fast save figure is quoted; the route back is not.
Restores share the path
Take an illustrative 4 TB checkpoint whose shared restore path sustains 100 GB/s of read bandwidth. Reading it alone takes at least 40 s. If 8 jobs, each restoring its own 4 TB checkpoint, start at once and share that path equally, each read takes 320 s — before capacity acquisition or state rebuild. Admitting them one at a time keeps the 320 s makespan but cuts the mean to 180 s, and the idle job-seconds from 2,560 to 1,440. The periodic model cannot price this case: a correlated incident breaks its independence assumption, and with K concurrent restores r becomes r(K). The 320 s is the read component of r conditional on that incident, not an input the formula covers.
Placement compounds it. The copies easiest to write may sit in the same failure domain as the failure, while the durable remote copy needs the slow path exactly when many jobs want it. In-memory and peer-replicated copies cut r only for failures their domain survives. A fast asynchronous save shrinks c; it says nothing about this contention, or about the lag between snapshot and durable completion.
Every second of restore is billed
During a restore the whole reserved job is billed and does no work. Idle cost per incident is roughly the number of accelerators times r times the hourly rate, and the expected idle share of the bill is simply λr — 2.96% of reserved spend at the illustrative rate above, for the read component alone. This is the price of access held idle, which is exactly what an hourly price measures (008); restore time sits inside the goodput accounting 005 describes. Storage, meanwhile, is usually bought by capacity, while a restore consumes bandwidth — the line where 001 found the storage premium sitting.
A recovery contract
Specify recovery before choosing checkpoint placement: which failures a copy survives, how freshness is established, what replacement topology is acceptable, and which resources a restore may reserve. Separate durable completion from background transfer still in flight. Include model state, optimizer state where needed, learning-rate schedule and step counters, data-loader position, random state and coordination metadata — CheckFreq checkpoints data-loader state precisely so each item is still used once per epoch after a resume.8 Set an explicit priority between restores and background saves, because one storage network cannot promise both unrestricted bandwidth, and admit concurrent restores deliberately rather than letting every job saturate the path at once.
Then test it: restore from the surviving copy into replacement capacity under realistic concurrent recovery, not by loading a local file into idle machines, and measure time to resumed useful work, including validation and warmup. A checkpoint is valuable because it limits lost work and restores service, not because it produced a high write bandwidth figure. The reliability budget that stops counting when the write returns is pricing the cheap half of the route.
References
- Eisenman et al., “Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models,” NSDI ’22. usenix.org
- Llama Team, AI @ Meta, “The Llama 3 Herd of Models,” §3.3. arXiv:2407.21783
- Wang et al., “GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints,” SOSP ’23.
- Wan et al., “ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development.” arXiv:2407.20143
- Young, “A First Order Approximation to the Optimum Checkpoint Interval,” Communications of the ACM 17(9), 1974.
- Daly, “A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps,” Future Generation Computer Systems 22(3), 2006.
- Benoit, Du, Herault, Marchal, Pallez, Perotin, Robert, Sun, Vivien, “Checkpointing à la Young/Daly: An Overview,” IC3 2022.
- Mohan, Phanishayee, Chidambaram, “CheckFreq: Frequent, Fine-Grained DNN Checkpointing,” FAST ’21. usenix.org
Provenance note, per house style. Illustrative: the 4 TB checkpoint, the 100 GB/s path, the 8 concurrent jobs, the one-failure-per-three-hours rate and the save costs. Computed on this page from them: the 40 s and 320 s reads, the 180 s staggered mean and idle job-seconds, the optimal intervals and the loss shares. Quoted: the Llama 3 checkpointing aim and interruption counts (ref 2), GEMINI’s exclusion of machine replacement (ref 3), Daly’s finding that the restart term does not move the optimum (ref 6), and CheckFreq’s data-loader checkpointing (ref 8).