Backup Planning for S3-Backed Workloads

Most teams think S3 durability means their data is backed up. It doesn't, and that gap is the subject of this piece. S3's eleven-nines durability promise covers hardware failure: disks dying, drives corrupting, racks going dark. It says nothing about someone with the wrong IAM permissions deleting a bucket on a Tuesday afternoon, which is a very different problem and the one that actually bites teams running AI workloads.

The backup model most companies still run was built for a world of slow, boring data. Financial records. HR files. Things written once and read rarely, cooling off in a predictable curve until they land in deep storage and stay there. Nightly backups, 90-day cold transitions, done. AI pipelines behave in an entirely different way. A training pipeline writes checkpoints every few hours, in large blocks, under time pressure. A retrieval system needs small reads back in milliseconds. A monitoring stack needs sustained, parallel reads during training and then total silence between runs. Putting sustained parallel reads, low-latency small reads, large checkpoint blocks, and immutable backup bursts all in the same bucket produces five different storage behaviors competing for one policy. A single nightly-backup rule can't serve all five. It was never built to.

The four artifact types in an S3 AI bucket

A typical machine learning bucket holds at least four kinds of objects, and each one ages on its own schedule. Treating them as one undifferentiated pile of "data" is where backup policy falls apart.

Raw training datasets get hit hard during an active run, pulled in parallel, read over and over. This quiet period looks like cold data to a standard policy, but it is dormant: a classic archive assumes cold data stays cold, while a dormant training set needs to come back fast the moment a new training cycle starts. Moving that dataset to Glacier on the usual 90-day inactivity rule brings the bill due exactly when nobody wants it, right as the next training run begins and retrieval fees stack up against the deadline.

Model checkpoints pile up fast while training runs, and most of them turn out to be worthless within hours of being written. But one checkpoint, maybe the one right before a production model started drifting, or the one tied to a regulatory request, becomes irreplaceable the instant someone needs it. The real question for checkpoint backup was always which checkpoints to keep, for how long, and how fast a team needs to get one back when something breaks.

Inference logs appear in huge volume with low individual value. Any single log line rarely matters on its own, but the aggregate becomes the signal a monitoring team needs to catch drift, or the fine-tuning team needs for the next round of training. Logs tend to grow faster than most teams can even review them, let alone process them for use.

Then there's everything else: annotation sets, versioned dataset snapshots, evaluation outputs, benchmark results. Nobody touches these for weeks or months at a stretch. Then a compliance audit lands, or someone needs to compare two model generations side by side, and these quiet files become the center of attention overnight.

S3 tier selection by access-frequency threshold

A simple temperature model maps each of these artifact types onto the right S3 tier, and it starts with naming what counts as hot, warm, and cold for AI work specifically, not for the compliance playbook most tiering guidance still borrows from.

Hot storage, meaning S3 Standard, holds whatever is actively in use right now: the training data feeding a live run, model weights currently serving production traffic, the checkpoint a team is actively validating. Cold storage, Glacier or Deep Archive, is for legacy corpora nobody's touched in a long while, retired checkpoints that have been fully superseded, and inference log batches with no recent activity.

The timing is where most tiering advice gets AI workloads wrong. Standard guidance says move to cold storage after 30 to 90 days of inactivity. That's a reasonable rule for financial records and nearly useless for AI artifacts, because a training dataset can sit untouched for four or five months and still come roaring back the moment a new model generation starts. A six-to-twelve-month inactivity window fits AI data far better than the 30-to-90-day default built for compliance archives.

Checkpoints deserve a specific call-out here because their access pattern is genuinely unpredictable. Explicit Glacier rules make sense only for artifacts with a known, fixed retention horizon, something like a compliance snapshot with a hard expiry date stamped on it. A checkpoint that might be needed at any time doesn't fit that mold, and forcing it into a fixed Glacier schedule just recreates the retrieval-fee trap from the dataset example above.

Getting this right matters for cost, not just directional accuracy. Warm storage runs roughly ten times the annual cost per terabyte compared to cold archival. At petabyte scale, that multiplier turns a small misclassification into a serious budget line, in either direction: leave too much in warm storage and the bill balloons, push too much into cold storage and retrieval fees eat the savings right back up.

Lifecycle policy design for AI pipelines without breaking the pipelines themselves

Lifecycle rules only do their job if the tagging underneath them is sound, so the real work of lifecycle design happens before a single rule gets written, when teams decide how objects get labeled. Tag by artifact type, not by bucket. An object tagged "active" should stay in hot storage no matter how old it gets, overriding whatever a time-based rule would otherwise do to it. An object tagged with a compliance snapshot identifier should route straight into Glacier the moment it's created, skipping the warm tier entirely because its retention horizon is already known and fixed.

A few traps appear repeatedly once teams start automating this, and they are worth planning around explicitly. Small files cause one of the costliest surprises. The fix is almost comically simple: bundle small files into tarballs before uploading them, so Glacier sees a handful of reasonably sized objects instead of thousands of tiny ones, and the minimum billable size charge stops being a tax on every single file.

Minimum storage duration charges are the second trap. Transition or delete an object before that window closes, and a prorated early-deletion charge appears on the bill: an easy way to quietly overpay for storage that looked cheap on paper.

Timing itself is the third trap, and it trips up pipelines that expect precision S3 doesn't promise. Lifecycle transitions typically land within about a day of their trigger date, not at the exact moment the rule fires. The mechanics behind all of this, Object Lock, versioning, lifecycle rules, work exactly as advertised. They're only as accurate as the tagging and classification feeding them. That's why the governance decisions have to come first and the automation second.

Credential compromise as the actual threat model for S3 backup

S3's durability number covers hardware failure. The actual danger to AI training data is someone getting into the account who shouldn't be there, and that's a threat replication and versioning inside the same account do nothing to stop.

Under the AWS Shared Responsibility Model, AWS covers the infrastructure, the physical durability, the uptime of the storage layer itself. None of it is covered by S3's durability guarantee, because durability was never designed to answer for human or credential failure.

The mechanics of a credential compromise make this concrete. Replication inside one account protects against a drive failing. It does nothing against a login getting stolen, because the attacker who has the keys to the front door generally has the keys to every room behind it, including the one marked "backup." A real backup has to sit outside that blast radius entirely, in an account the compromised credentials can't reach.

Object Lock exists for exactly this kind of protection, immutable retention that even a holder of valid credentials can't undo before a set retention date expires. But immutability isn't a single switch that's either on or off. It only does its job when the lock sits on infrastructure architecturally separated from the production credential surface: the account and the keys used for day-to-day pipeline work. A backup locked down inside the same account as production data is still reachable by whatever compromised the production account, lock or no lock.

The checkpoint recovery SLA problem

A checkpoint backup plan that doesn't name a retrieval latency target is a storage policy wearing a recovery plan's clothes, and the gap between the two becomes visible the moment a rollback is needed and the GPUs sit idle waiting for a file to come back.

Checkpoint reads happen rarely but matter enormously when they do. A rollback, a redeployment, a regulatory request, any of these can call for a specific checkpoint version with no advance warning, including one that hasn't been touched in months. That read path needs to be ready at all times, not just when it's convenient.

The tier a checkpoint sits in determines how long that wait actually runs. Deep Archive stretches further still, up to 48 hours depending on which retrieval tier gets selected. For a checkpoint needed to restart a training run that just failed, that window means GPU time sitting idle, burning money, while the file makes its way out of deep storage.

The write side of checkpoint storage and the read side are optimizing for entirely different things. Writes need to absorb a burst, hundreds of gigabytes landing at once, without creating backpressure that slows down the GPU layer doing the training. Reads need to deliver the correct checkpoint version at whatever latency the recovery plan actually requires. Defining a recovery time objective for each artifact type, rather than one blanket number for the whole bucket, is the direct payoff of building the artifact taxonomy. A production checkpoint needed mid-rollback usually can't, and the tier choice for that checkpoint has to reflect the recovery SLA, not just how often the file gets touched.

Cross-account and cross-cloud backup architecture for workloads that span providers

AI infrastructure increasingly runs disaggregated, with GPU compute sitting at one provider and object storage sitting at another. That split makes cross-account backup architecture more necessary and considerably more expensive to get wrong.

Backblaze B2 is one example of storage built with this split in mind, offering S3-compatible storage with egress-free access from select partner GPU and HPC providers. Pairings like this matter because retrieval costs can otherwise turn a cross-provider setup into a budget problem on their own, independent of the actual storage bill.

The account-separation argument from the previous section carries over directly here. A backup sitting in the same account as the primary data offers no protection against a credential compromise in that account, no matter which provider holds the bytes. The requirement is separation at the account level, not just a different bucket name.

The S3 API itself makes this kind of cross-provider setup workable without custom engineering for every pairing. The same API contract runs against multiple storage providers. A backup copy can live with a different vendor than the primary data without a bespoke connector built just for that link. Azure Blob Storage is the exception: it doesn't natively support the S3 API and needs a third-party gateway to bridge the two. Storage built to mount the same way against multiple backends, without a migration project or a custom ETL pipeline for every new provider, lets a team move backup copies across providers without rebuilding how the pipeline reads and writes data every time infrastructure shifts. That's the practical argument for storage that doesn't lock a team into one provider's walls.

Building the artifact-aware backup policy: a practical decision framework

A sound backup plan for S3-backed AI workloads comes from mapping each artifact type to four decisions, then writing the lifecycle rules and Object Lock configurations that match what those decisions call for.

The first decision is retention horizon: how long a given artifact needs to stay recoverable, a number fixed by a compliance requirement or set by operational need, in which case it should get estimated on the conservative side. The second is recovery time objective, set per artifact type rather than per bucket: a raw dataset can often tolerate a multi-hour wait, while a production checkpoint needed for an active rollback usually cannot, and the tier choice should follow that distinction. The third is immutability mode, meaning whether an artifact needs Object Lock protection at all, and if so, whether that lock sits on infrastructure genuinely separated from the production credential surface, since a lock inside the compromised account protects nothing. The fourth is account isolation: whether the backup copy for a given artifact type lives in a separate account, and ideally with a separate provider, so that a credential compromise in the primary account cannot reach it.

Running every artifact type, raw datasets, checkpoints, inference logs, annotation and evaluation sets, through these four questions produces a policy that looks nothing like a single nightly-backup rule applied uniformly across a bucket. It looks like four or five distinct configurations, each one matched to how that artifact actually behaves, which is the entire argument this piece has been building toward from the first paragraph.