S3 as an AI Training Data Store

S3 holds more than 500 trillion objects across hundreds of exabytes. That's the reflex answer to "where do we put our training data," and it's a fine one, right up until you try to actually feed that data to a GPU. Storing data and feeding a model are two different jobs, and S3's API was built for the first one. Every cache layer, custom loader, and preprocessing service teams bolt onto their buckets traces back to that single gap.

So why did S3 get this big in the first place? Turns out size isn't just marketing here.

More than a million data lakes run on AWS. Object storage grew up alongside that pattern because it scales to petabytes without the namespace headaches that choke a normal filesystem past a certain size. Metadata lives right on each object, so tagging and filtering a training set doesn't need some separate system bolted on the side. Durability on S3 Standard is exceptionally high, geo-redundancy comes standard, and nobody has to commit to capacity up front. You pay as the dataset grows, which is a nice change from the old world of buying disks you hoped you'd fill.

The API itself outgrew S3. The S3-compatible object storage software market hit $6.7 billion in 2024 and is headed toward $23.4 billion by 2033, growing 15.2% a year. It became a universal contract, the way SQL did for databases decades back. And the money behind it is real: AI-related cloud spend went from 8% of total cloud spend in 2023 to a projected 19% by 2026, storage riding along as one of the fastest-growing line items on the bill.

S3 won on economics, on durability, on sheer ecosystem breadth. That much isn't up for debate. But winning the storage argument never meant winning the training argument, and that gap is what the rest of this piece digs into.

What S3's own API was actually designed to do

S3 is an object store. It works through PUT, GET, and DELETE against a flat namespace of keys. Not a filesystem, never tried to be one.

The design leans toward durability and availability over speed, because the jobs it was built for were backups, archives, and the occasional big read. Each request stands alone. There's no seek position, no directory to walk down, no sense that request 400,000 has anything to do with request 399,999.

Training breaks every one of those assumptions at once. A training run does the same sequential read over the same big dataset, over and over, epoch after epoch, nothing occasional about it. It also means millions of small reads, image crops, tokenized text shards, where the per-call overhead piles up fast. And checkpointing means periodic big writes where any added latency stalls the entire run, GPUs included.

The cost model gives away that AWS knows this. Fewer, larger objects mean fewer LIST and GET calls, which is exactly why AWS bumped the max object size from 5 TB to 50 TB in 2025. This isn't about fitting bigger files. It's about letting teams restructure data so a job makes a few thousand calls instead of a million.

None of this makes S3's API broken. It does what it was built to do. AI training just turned out to be a genuinely different animal than backups and archives ever were.

Where native S3 latency actually lands, and what that costs a training run

S3 Standard latency runs 50 to 150 milliseconds per object request, sometimes worse. Fine for a data lake query you run twice a day. Brutal once you multiply it across millions of training samples an epoch.

Here's the expensive part. GPUs are the priciest line item in your stack, and when storage can't keep pace with how fast a GPU wants to eat data, the GPU just sits there idle, waiting on a delivery stuck in traffic. Industry benchmarks treat high GPU utilization as the baseline expectation; when sustained utilization drops well below that, storage is almost always the culprit. Even hyperscalers with full engineering organizations behind them have documented clusters where significant GPU cycles sat stalled waiting on training data.

It's not just the big players either. MinIO's research found more than half of organizations report data and storage bottlenecks limiting their AI performance. Everybody's got the same problem, just at a different scale.

Checkpoint writes make it worse. Cloud object storage caps throughput per bucket within a fairly narrow band, and at scale, checkpoint operations can stretch into minutes. Dead time on the GPU meter, running the whole while.

A storage bottleneck never shows up on the storage bill. It shows up in GPU hours, and GPU hours cost orders of magnitude more than the storage sitting quietly underneath them.

What AWS built to close the gap from within S3

AWS noticed. They've spent the last couple years building tools to close this gap without leaving S3 itself.

S3 Express One Zone is the headline act. Latency drops to 5 to 10 milliseconds against 50 to 150-plus on Standard, up to 10 times faster. In benchmarks downloading 100,000 objects, Express hit roughly 9 GB/s against Standard's roughly 1 GB/s. A 4 KB object download that averaged 19ms on Standard dropped to 3.8ms on Express, an 80% cut. It supports up to 2 million reads per second and 200,000 writes per second per directory bucket. Teams migrating over have cut epoch times by up to 35% without touching a line of model code, which is about as close to a free lunch as storage gets.

None of it comes free, and AWS doesn't pretend otherwise. Per-GB cost on Express runs roughly 7 times higher than Standard, which rules it out for a full dataset at any real scale. Durability drops to 99.9% in a single availability zone against Standard's exceptional durability guarantees, an honest trade of safety for speed. Single-AZ placement also means it's not a general-purpose store. Instead, it's a hot tier that sits next to your source of truth, not instead of it.

The rest of the toolkit fills in the edges. S3 Batch Operations now runs up to 10 times faster and scales to 20 billion objects per job, useful for the preprocessing that happens before training even starts. S3 Vectors now handles up to 20 trillion vectors per bucket, a 40x jump from preview capacity, with storage and query costs cut by up to 90%, AWS stretching the S3 interface toward AI-native data structures beyond plain files. S3 Tables, with Apache Iceberg support, brings ACID transactions, schema evolution, and time travel to object storage: the lakehouse pattern now living natively inside S3.

Add it up and you get AWS tiering and specializing, not rewriting the physics underneath S3 Standard. Every one of these products is a quiet admission that the base latency gap is real. AWS didn't close it evenly, choosing instead to build specific tools around specific pieces of it.

The architectural patterns teams build when S3 alone is not enough

Nobody ditches S3 over this. Instead they build an access layer, some kind of translator sitting between the bucket and the compute, turning the object API into something a training framework can actually chew on.

A handful of patterns show up again and again, each with its own catch.

Local SSD caching, pre-downloading the whole dataset onto instance-local NVMe before the job starts, kills per-request latency during training itself. But now there's a waiting phase before any GPU does real work, and the cache goes cold the second the run ends. FUSE-based S3 filesystems (mountpoint-s3, s3fs) make the bucket look like a normal file path, convenient right up until you hit their limits: most skip full POSIX semantics, atomic rename, flock, mmap, so existing training code breaks at exactly the edges nobody tested. Custom data loaders built into PyTorch's DataLoader or tf.data hide latency through prefetching and parallelism, but that's per-framework engineering work, and it does nothing for checkpoint writes going the other direction. Preprocessing pipelines that convert raw S3 objects into WebDataset, TFRecord, or Parquet shards genuinely help throughput, at the cost of duplicated storage, a pipeline somebody now has to maintain forever, and a lag before new data is actually usable.

Some hyperscalers have built internal data preprocessing services specifically to kill data stalls at petabyte scale. These work well, though they represent a massive engineering investment that basically nobody outside a handful of hyperscalers has the headcount to copy.

MinIO's research puts a number on the underlying condition: 57% of enterprises say their data isn't AI-ready. A good chunk of what "not ready" means, in practice, is nobody's built the access layer yet. That's not a checkbox on a slide. It's real engineering surface area, real latency between when data lands and when a training job can touch it, and one more system somebody has to keep alive at 3 a.m.

What a filesystem interface changes about feeding a training run

Training frameworks, shells, and AI agents all speak one native language, and it's the filesystem: paths, reads, writes, seeks, directory listings. That's what they grew up on.

Not a coincidence, not a taste preference. Frontier models train heavily on bash and file manipulation, and agents reach for filesystem calls almost by reflex when given the choice. That preference isn't some UX layer bolted on afterward. It's baked into the model weights themselves.

A POSIX filesystem layer sitting on top of S3 changes the equation in one specific way: it translates GET and PUT into open, read, and write, so existing training code runs as-is, no SDK rewrite required. An NVMe cache sits on the read path, sub-millisecond on a hit, falling back to the bucket only on a miss. The bucket stays the source of truth the whole time; nothing gets copied away. It just becomes reachable through a different door.

The POSIX details matter more than they sound like they should. Checkpoint patterns in PyTorch and JAX rely on writing to a temp file and renaming it atomically. Skip that step and an interrupted checkpoint corrupts, full stop. Distributed jobs coordinate through flock and fcntl; without real locking, multi-node training quietly drifts out of sync, and nobody notices until the loss curve looks wrong. Memory-mapped reads, common for large-file random access, need actual kernel-level support, not a FUSE layer just waving calls through. Dataset versioning tools assume hard links and symlinks behave the way they're supposed to.

Here's the line that actually separates a real filesystem layer from a FUSE mount that's just repackaging S3 calls: a real layer with NVMe caching absorbs that 50 to 150-millisecond round trip at the cache itself. The application never sees the object store's latency. It just sees a filesystem, acting like a filesystem, which is the whole point of the exercise.

Agents make this even more important. A persistent filesystem workspace lets an agent read, write, and run commands against the same data across sessions and parallel runs, something neither a plain object store nor a throwaway scratch volume can give it.

How Archil fills the access-layer role without replacing the bucket

Archil's pitch is direct: point it at an existing S3, GCS, R2, Azure Blob, or other S3-compatible bucket, and it mounts as a real POSIX filesystem. No migration, no ETL step, no rewriting training jobs or agent code to match some new SDK.

Reads hit an NVMe cache at sub-millisecond speed. On a miss, Archil fetches from the source bucket and caches it, so the 50 to 150ms S3 Standard penalty gets absorbed at the cache layer instead of landing on the training job itself. Writes replicate first, then flush asynchronously back to the bucket, so checkpoint writes don't sit around waiting on an object-store round trip. Capacity scales elastically, and billing tracks only what's actually cached, so nobody has to guess up front whether a job needs 10 GB or 10 TB.

The bucket stays the source of truth throughout. Archil never keeps a persistent copy of your data outside your own account, and access can be revoked any time, since the data never actually left home. That's a direct answer to the data residency worry that makes a lot of teams nervous about adding any caching layer at all. Nothing moves cross-region unless someone explicitly sets it up that way.

It's cloud-agnostic on purpose, too. S3, GCS, R2, and Azure Blob all mount the same way, so teams stop hand-building separate data-movement plumbing for every cloud they happen to train in. On the compute side, Archil runs serverless execution right alongside the filesystem, so an agent runs commands directly against its own files without some separate sandbox bolted on elsewhere. Compute attaches to the filesystem as a service, takes bash commands, returns results. Same design idea the last section was building toward.

Archil doesn't replace S3. It's the access layer the previous section described, built so teams stop reinventing the same caching and mounting setup right before every single training run.

Choosing the right storage tier for each phase of a training pipeline

A training pipeline has distinct phases, and each one wants something different from storage. No single tier serves them all well, and pretending otherwise is how teams end up either overpaying or staring at stalled GPUs. Sometimes both.

For the raw data lake, the actual source of truth, S3 Standard is still the right call. Durability, cost, ecosystem integrations, no lock-in to worry about. The jump to 50 TB max object size makes it easier to keep datasets as a manageable handful of large objects instead of millions of tiny ones scattered across the namespace.

For active training reads, the answer splits depending on setup. If the workload lives entirely inside AWS and the budget has room, S3 Express One Zone works well as a hot tier; the 7x cost premium is easy to justify for data a running job is actively chewing through, much harder to justify for an entire archive sitting mostly idle. If the workload spans multiple clouds, or the team is simply tired of hand-managing data movement before every run, a filesystem access layer like Archil, caching straight from the bucket and exposing POSIX on top, gets the job reading at NVMe latency no matter which cloud the bucket actually lives in.

For checkpointing, the requirement is blunt: writes need to land fast enough that the GPU isn't sitting there burning money waiting on a save to finish. Whichever tier handles this needs asynchronous flushing back to durable storage, not a synchronous wait on every single write.

None of these choices are permanent, either. Datasets move between tiers as they move through the pipeline, hot today, cold next month. Teams that plan for that movement upfront spend a lot less time firefighting six months later.