S3 as an AI Training Data Store

S3 won the AI storage race for structural reasons, not habit. Three design properties line up with how training and inference actually spread across machines. The API is stateless and scales horizontally, which matches the way training jobs and inference requests fan out. It also offers a contract big enough to run real production work: PUT, GET, LIST, multipart upload, object lock, lifecycle rules, versioning.

File storage and block storage didn't lose this fight because of price or politics. They lost on object count. A modest training corpus holds tens of millions of files. A production retrieval index holds hundreds of millions. Try pointing a regular filesystem at that and directory traversal falls over long before you get there. Block storage has a different problem: it can't hand the same dataset to thousands of compute nodes at once without some coordination layer standing in the middle, and that layer becomes the new bottleneck it was supposed to prevent.

The scale involved is hard to picture, so here's a comparison: AWS S3 now holds more than 700 trillion objects across hundreds of exabytes of storage, processing hundreds of millions of requests every second on average. That scale shows how thoroughly the rest of the industry has settled on this one interface as the default, not just AWS flexing.

AWS keeps building features squarely aimed at AI workloads, which tells you where the puck is going. S3 Express One Zone is now the fastest storage class AWS offers, S3 Tables bakes in native Apache Iceberg support, and S3 Vectors handles vector storage and search natively. Late in 2025, AWS also raised the maximum object size from 5 TB to 50 TB, and that cleared out a ceiling that used to give teams storing high-resolution video or seismic survey data real headaches. None of this reads like a storage vendor patching holes. It reads like a company building toward a specific kind of customer.

Where the object interface creates friction at training time

The same qualities that made S3 the obvious choice also make it a rough feed straight into a GPU. Object storage was built for durability and broad accessibility, not for the sustained sequential throughput or the low-latency random reads a training loop needs minute to minute. The thing that made S3 universal is the same thing that makes it slow in this one specific job.

Picture a GPU as a very expensive, very impatient line cook. Two distinct approaches are converging on eliminating the network-and-CPU overhead between S3 and GPU memory: RDMA at the protocol level, and compute-local NVMe caching at the product level, so if the ticket printer takes too long to spit out the order, the cook just stands there. When a distributed training run spans dozens of nodes, those idle stretches don't stay small. They stack up, and GPU utilization drops hard. One practitioner account describes a transformer training script pointed directly at S3, GPU utilization cratering, and a training run that took weeks longer than it should have. It's what happens the first time a team treats object storage like a hard drive.

A second trap waits in small-file datasets: it doesn't appear on a bandwidth chart. The math on paper says there's more than enough throughput available. In practice, the storage layer burns most of its time just finding and opening each tiny file, not reading the bytes once it gets there. Buying more bandwidth doesn't fix this, because bandwidth was never the problem. The lookup was.

Both of these failure modes trace back to the same mismatch. The object interface was built around arbitrary-sized blobs addressed by key, but a training loop needs a librarian who already knows the shelf, not a search party sent out every time someone asks for a book. Teams don't avoid this by being careful. They hit it on the first real training run, because it's baked into what object storage was designed to do.

The three-tier architecture that production teams use to close the gap

The fix that's taken hold across the industry is a layered system: hot data staged on local NVMe right next to the GPUs, warm data sitting in object storage with prefetching running in the background, and cold data parked in cheaper archival tiers. As of mid-2026, this three-tier pattern is the verified standard teams build around, not an experimental workaround.

The logic behind each layer is simple once you see it laid out. NVMe handles the tight, repetitive read loop that a training job runs thousands of times an hour. Object storage handles durability, bulk capacity, and the original copy of the dataset. The real engineering work happens in between: getting data staged onto NVMe before the job actually needs it, and getting checkpoints flushed back to the bucket without freezing the training loop while it waits.

This handoff isn't a nice-to-have bolted on for speed. On a reference 8× A100 cluster, pre-staging data onto NVMe eliminates enough wasted I/O time to save approximately $2,500 a month in compute costs. The staging layer pays for itself out of the compute budget it protects.

The real question for any team planning storage for AI work was never which bucket to buy. It's how data moves from that bucket to the GPU and back again without the GPU ever noticing the trip. A bucket is necessary. It is not sufficient. If a team wires a training job straight into S3 and calls it done, they haven't made a configuration mistake. They've made an architectural one.

RDMA and compute-local caching are changing what "fast enough" means for object storage

Two different engineering bets are chasing the same target: cutting out the network and CPU overhead that sits between S3 and GPU memory. One works at the protocol level, through RDMA. The other works at the product level, through caching that lives right next to the compute.

On the protocol side, NVIDIA is pushing remote direct memory access as a way to get around the old object-storage speed ceiling entirely. Instead of routing data through TCP and taxing the host CPU along the way, RDMA lets data move directly between storage and GPU memory. Cloudian, Dell, and HPE are already building this into real products, including HyperStore, ObjectScale, and Alletra Storage MP X10000, and NVIDIA is working with those partners to push the approach toward a shared standard, arguing accelerated object storage is no longer optional for large-scale AI systems. SNIA is doing parallel work standardizing S3 transfers over RDMA for direct GPU memory access, which is a signal that this has moved past vendor marketing and into genuine industry infrastructure.

On the product side, CoreWeave's AI Object Storage, known as CAIOS, shows what compute-local caching looks like in practice. Announced at NVIDIA GTC and reaching general availability in March 2026, its Local Object Transport Accelerator (LOTA™) caches the datasets a job uses often and prestages data directly onto the local NVMe disks sitting on the GPU nodes themselves, cutting network latency and speeding up training. Benchmarks put CAIOS at 2 GB/s per GPU, a number that holds up scaling out to hundreds of thousands of GPUs. A March 2026 update added proactive cache warming, so if a job starts with a warm cache, its first epoch runs noticeably faster than one reading cold.

Neither of these approaches touches the S3 API itself. Both accelerate it rather than replace it. Pipelines don't get rewritten, object storage stays the source of truth, and the only thing that actually changes is the path data takes between the bucket and the GPU. The API has turned out to be remarkably durable.

A POSIX filesystem mounted over object storage: a different answer to the same problem

RDMA and compute-local caching both leave the object API in place and speed up the road leading to it. A different camp argues the road was never the real problem: the object API itself is the mismatch, and the fix is to mount object storage as a POSIX filesystem so training code never has to know S3 is back there at all.

The argument holds up once you look at what most ML tooling actually expects. Shell scripts, data pipelines, and a large share of existing training code are written against filesystem operations: open, read, write, seek, rename. A POSIX mount lets that code run as-is, with nothing rewritten, once it replaces the interface.

A POSIX mount only helps if the "POSIX" it implements is the real thing. Atomic rename, flock and fcntl, mmap, fsync, hard links, symlinks, sparse files, these aren't extra features to tack on later. Training loops lean on atomic writes to protect checkpoints, data loaders use mmap for zero-copy reads, and dedup tools depend on hard links. Missing any of these does not throw an obvious error in the system. It breaks quietly, which is worse.

Under the hood, the performance model mirrors the three-tier pattern from earlier in a tighter package. Reads that hit the NVMe cache come back in under a millisecond. A cache miss fetches from the source bucket and caches the result for next time. Writes get replicated before the system confirms them, then flushed back to the bucket asynchronously. The bucket stays the single source of truth, and no copy of the data lives anywhere outside the user's own account.

This approach also doesn't care which cloud is underneath. The same mount works against S3, GCS, R2, and Azure Blob with no migration, no ETL step, no code changes. Teams don't end up swapping one form of lock-in for a different one wearing a filesystem costume.

There's a newer argument for this approach that has nothing to do with legacy scripts. Giving agents a plain bash interface instead of a custom connector for every data source cuts down on wasted context and matches how the model already thinks about files.

What S3 compatibility guarantees

The S3 API itself really is a cloud-agnostic layer, as far as it goes. The trouble starts at the feature layer sitting just above it. Lifecycle rules, Object Lock modes, S3 Select, bucket logging, each vendor implements these a little differently, so any pipeline leaning on them doesn't move cleanly from one provider to another.

If teams want to move freely between providers, they need to build against the common ground of the S3 API itself and pair that storage with an open table format like Apache Iceberg or Delta Lake, so metadata travels with the data instead of getting stranded when storage moves. Skipping that step makes portability a promise rather than a plan.

The Airbus case shows what the same trap looks like one layer up, on the compute side rather than storage. Airbus engineers found that AWS's specialized AI chips locked them into proprietary APIs and toolchains, and moving even a single model to a different chip meant rewriting large chunks of the ML pipeline. S3-compatible storage can't fix compute lock-in on its own, but it keeps the data side open while you sort out the compute side.

Self-hosted S3-compatible stores exist for teams that want that openness but don't want to hand control to any single cloud. Ceph RGW, SeaweedFS, Scality RING, and Cloudian HyperStore all offer the same API surface across on-premises, edge, and multi-cloud setups. SmartMore runs this exact pattern at petabyte scale for AI training data, and Scality RING XP reaches microsecond response times even on small-object workloads. Hybrid deployment is picking up steam for a reason that has nothing to do with performance: organizations handling data under sovereignty rules need storage running on their own premises while still speaking the same API their cloud-based AI tools expect, and S3-compatible self-hosted stores deliver that without bolting on a separate abstraction layer.

Choosing among the main S3-compatible providers for AI

Picking a provider comes down to matching the workload to what each option actually optimizes for, not chasing whichever name is loudest that quarter. A hyperscaler bucket makes sense for deep framework integration, managed table formats, and features like S3 Express One Zone or native vector storage that smaller providers simply haven't built yet. That convenience comes bundled with tiered pricing and extra fees that get harder to predict once a workload scales up, so it suits teams that value speed of setup over cost control.

Infrastructure providers built around AI workloads specifically, in the mold of CoreWeave's CAIOS, make a different trade. CoreWeave's AI Object Storage (CAIOS), reaching general availability in March 2026, caches frequently used datasets and prestages data directly on local NVMe disks of GPU nodes to reduce network latency and improve training speeds, and a March 2026 update added proactive cache warming so training jobs starting with a warm cache see a materially faster first epoch compared to cold reads.

Self-hosted, S3-compatible software is the right call for teams that need the API without needing the cloud underneath it. Ceph RGW, SeaweedFS, Scality RING, and Cloudian HyperStore all let a team run the same interface every tool already expects, on hardware the organization controls. That's the natural choice for sovereignty requirements, edge deployments, or any workload that can't leave the building for legal reasons rather than technical ones.

None of these options are mutually exclusive, and most serious AI infrastructure ends up blending them: a hyperscaler bucket as the durable source of truth, a caching or RDMA layer to feed GPUs without starving them, and in regulated cases, a self-hosted store running in parallel to satisfy a compliance team that doesn't care how fast the training run goes. What a team builds on top of it is the actual decision.