Checkpoint Storage Patterns for Distributed Training
Faster checkpointing patterns preserve GPU utilization when distributed training clusters fail.
Distributed training jobs fail. Constantly, and at scale, with numbers big enough to make finance people spill their coffee. Checkpointing is the only thing standing between one of those failures and losing days of GPU time you already paid for.
During Llama 3 pre-training, Meta logged 466 job interruptions over a 54-day stretch. 419 of those were unexpected, and 78% traced back to confirmed or suspected hardware problems. That's a routine occurrence, not a rare edge case you patch around once and forget. That's Tuesday. And because distributed training runs synchronously, one GPU dying doesn't just take out one GPU: it stops the entire cluster, every GPU in it, waiting on a node that's gone quiet. Despite all that, Meta still pulled off better than 90% effective training time at 16K-GPU scale. That number didn't happen by accident. It happened because someone designed the checkpoint architecture on purpose, instead of trusting the defaults.
How checkpoint frequency and storage speed determine how much compute you get
Checkpoint too rarely, and one failure wipes out hours of work. Checkpoint too often, and slow storage taxes you on every single write, bleeding compute a little at a time instead of all at once.
Run the math on Meta's failure rate, about 8.6 interruptions per day. Checkpoint every two hours, and a failure costs you 8.6 hours of retraining, which nets out to 64% goodput. That's a lot of processing time spent typing the same sentence twice. Dropping the interval to every 30 minutes brings wasted retraining down to 2.15 hours, pushing goodput to 91%. Same cluster, same failure rate, wildly different bill.
The real target sits further out: checkpoint every 4 minutes, with the write itself completing in about 2.5 seconds, dropping the loss in compute utilization to roughly 1%. That's the floor everyone's chasing. Getting there means the storage system has to absorb a full checkpoint write in seconds, not minutes. Which is, conveniently, the whole architectural puzzle this piece is about.
Why object storage alone cannot serve as a checkpoint target for large clusters
Object storage, S3, GCS, Azure Blob, is exactly where model artifacts should live long-term. It's durable, it's cheap, and anyone with the right credentials can grab a file from anywhere. No argument there.
The trouble is it was never built to catch a burst of checkpoint writes, and the way it fails is specific. A single S3 PUT runs 30 to 40 milliseconds, which is nothing for one file going up sequentially. But FSDP training doesn't write one file. It writes hundreds or thousands of shards at once, one per GPU process, all hitting the object store in the same instant. Every one of those writes sits in that 30 to 40 millisecond queue, and every GPU in the cluster idles in lockstep waiting for the slowest one to clear.
Pushing harder means hitting a second wall: S3 rate limits. Enough simultaneous ranks slamming the bucket at once can trigger HTTP 503 errors, which kick off client retries, which stall every GPU in the job for however many seconds it takes the backoff to resolve. Object storage is just answering a question it was never asked to answer this fast, this many times, all at once. It's just answering a question it was never asked to answer this fast, this many times, all at once.
The async offload pattern: writing to local memory first, flushing to durable storage after training resumes
The fix, in its simplest form: stop making the GPU wait on the slow part. Save the checkpoint to CPU RAM or local NVMe on the training node first, let training resume immediately, and flush that snapshot to durable storage in the background while the next step is already running.
The stall visible to the accelerator shrinks down to whatever it takes to copy model state out of GPU memory and into CPU RAM. Often a few seconds, instead of minutes. That's the whole trade: move the slow part off the critical path.
Except moving the risk off the critical path doesn't make it disappear, it just relocates it. If the node dies before that background flush finishes, the in-memory checkpoint goes with it, and recovery falls back to whatever the last durable checkpoint was. Say the node fails partway through flushing 350 GB to S3. That checkpoint never existed as far as durable storage is concerned. Sizing the buffer and picking the flush frequency is a running bet on how much data you're willing to lose if the timing goes wrong. It's a running bet on how much data you're willing to lose if the timing goes wrong.
The write-back caching pattern: a fast intermediate layer absorbs the write, object storage receives it later
A second approach puts a fast, POSIX-accessible caching layer, usually NVMe-backed, between the training cluster and the object store. Checkpoint writes land on the cache at local storage speed, not S3 speed, and the cache handles flushing to the bucket on its own schedule, asynchronously.
From the training job's point of view, this looks like writing to fast local disk: sub-2ms P99 latency, full local throughput, no waiting around for a bucket API to respond. Durability arrives once the cache pushes the data to the object store, but the GPU never sits around watching that handoff happen.
A POSIX write cache layer benchmarked at 7.6 GiB/s, against the 30 to 40 millisecond latency object storage delivers on a direct write. That gap is the entire argument for this pattern. And because the interface is just a filesystem, the training code doesn't know or care that it's talking to a cache instead of a local disk. Same file path, same write call, no rewrite required.
The staged storage tier pattern: hot local NVMe, warm parallel filesystem, cold object storage
Not every checkpoint deserves the same storage. A checkpoint from ten minutes ago and a checkpoint from three weeks ago are both technically "backups," but only one of them is likely to get read again soon. Staged tiering builds storage around that fact.
Three tiers, each doing a different job:
Hot tier: local NVMe or a high-performance parallel filesystem, holding the most recent handful of checkpoints. Written at training speed, read back at recovery time with almost no delay. Warm tier: a parallel filesystem like Lustre or IBM Storage Scale (formerly GPFS), or a comparable high-performance managed system. Holds checkpoints just past the hot window and supports several nodes reading at once. This is the tier that actually handles most real-world recovery events. Cold tier: object storage, for the checkpoints and model versions nobody needs today but somebody might need eventually. Optimized for durability and cost, not speed.
The numbers back this up. IBM Storage Scale hit 656.7 GiB/s read and 412.6 GiB/s write in MLPerf benchmarks against a 1-trillion-parameter checkpoint. For Llama 3.1 at that same 1T scale, load time landed around 23 seconds and save around 37 seconds. Lustre plays the same warm-tier role elsewhere: Lustre plays the same warm-tier role in other production ML platforms where storage performance directly affects GPU utilization and training step consistency. Once you can see uneven step times in your logs, that's storage waving a flag at you.
GPUDirect Storage and how hardware-level I/O acceleration changes the pattern calculus
GPUDirect Storage removes the CPU from the loop. Data moves straight from NVMe or network storage into GPU memory, skipping a copy step that would otherwise eat CPU cycles and PCIe bandwidth for no good reason.
GPUDirect Storage 2.0, available with CUDA 12.3 and up, delivered a 15% throughput improvement over previous versions. PCIe Gen5 NVMe drives push 14 GB/s per drive, which stacks up to 400 GB/s or more across a system. Early numbers on NVIDIA's Magnum IO running on Blackwell hardware show 250 GB/s sustained. Major cloud providers have moved to offer GPUDirect Storage-enabled instances tied into their block storage offerings.
So does this replace everything above it? Not quite. When one server can move 400 GB/s on its own, the case for a write-back cache on that single node gets thinner. But training doesn't happen on one node. It happens on hundreds of ranks writing at once, and their combined demand still outruns what any single machine's local NVMe can absorb by itself. GPUDirect Storage raises the ceiling. It doesn't change the fact that the ceiling gets tested by hundreds of GPUs at the same time, not one.
Network volume versus local NVMe: the default storage attachment decision that determines checkpoint survival
A decision teams make early and then forget about causes all of this: what kind of disk is the checkpoint actually landing on. That choice quietly decides whether a checkpoint survives a failure or evaporates with it.
Three attachment types, three very different survival stories. Container or ephemeral disk gets wiped the moment the worker stops, so anything written there survives nothing, not a crash, not a restart, not a scale-down event. That's the option that feels fine right up until the moment it doesn't. Local NVMe attached to the instance survives a process crash but not an instance termination, since the drive goes away with the machine. A network-attached volume, detached from the compute instance's own lifecycle, is the one that survives the instance disappearing, because the data was never tied to that instance to begin with.
None of these choices is universally correct. They trade speed for survival in different amounts, and the right pick depends on how much a lost checkpoint actually costs a given job. But defaulting to whatever storage type a cloud console pre-selects, without asking what happens when that specific instance dies, is how "checkpointing" quietly becomes "checkpointing, except for the part that mattered."
