High-performance object storage providers for AI data pipelines

Most teams building AI training infrastructure spend big on GPUs and treat storage as an afterthought, so they find out mid-training that the chips are sitting there waiting for data that hasn't arrived yet. That finding usually lands after the cluster is already provisioned, which is the expensive time to learn it. A 2026 analysis by Paul Speciale found that 57% of enterprises now rank storage performance as their top priority for avoiding AI bottlenecks, ahead of compute and GPU availability. That's a flip from the usual assumption that GPUs are the scarce, precious resource and everything else is plumbing.

The plumbing is where the trouble lives. Training jobs running on traditional enterprise storage, think NFS shares, standard S3 buckets, spinning disks, routinely get a fraction of the GPU utilization that the hardware is actually capable of. The GPUs are fed through a pipeline with several stages: ingestion, preprocessing, shuffling, checkpointing, retrieval. Each of those stages throws its own I/O demands and metadata lookups at the storage system, and general-purpose object storage was never built for any of those jobs.

Idle GPU time appears at the batch boundary. The model finishes a batch, and then it waits for storage to hand over the next shard of data. That wait repeats at every batch, every epoch, for the full length of a training run, so a small delay multiplies into a large amount of wasted compute time over the life of the job. Most teams building AI training infrastructure overspend on compute and underplan for storage, then discover mid-training that their GPUs are starving for data. The hardware's capable of far more than the errand it's running.

Standard object storage design under AI workloads

Object storage earned its place in the data center because it's cheap, durable, and it can hold enormous amounts of data without falling over. That was the job it was built for, and it does that job well. Training pipelines ask for something different: high throughput, low latency, and the ability to handle thousands of simultaneous requests for metadata without choking.

Object storage uses a flat namespace, so it can scale to billions of items without hitting the slowdown that hierarchical file systems run into. Hierarchical file systems, where everything lives in a tree of folders, start to choke on metadata lookups as the data gets more complex. Text files are simple to locate. High-resolution video, 3D point clouds, and real-time sensor telemetry are not, and hunting through a folder tree for the right file becomes a drag on an entire training epoch. Object storage sidesteps that specific headache. It doesn't solve the next one.

The next one is latency. Object storage trades the metadata tangle of file systems for a different ceiling: standard S3-compatible storage has a latency floor that's higher, and less predictable, than active training workloads need. Treating object storage as a universal hot tier, the single place everything lives while GPUs are actively training, is a mistake nearly every team makes the first time they build one of these pipelines.

The reason that mistake is so common is that an AI pipeline isn't one job with one storage need. Omdia's model breaks it into three stages, and each one has a different set of requirements. Ingest and curation care about capacity and cost. Data staging and active model training care about throughput and low latency. Production deployment cares about resilience. One tier of storage, however good, isn't built to serve all three jobs equally well.

Checkpointing is where the latency gap turns into a line item. Training a large model means periodically writing its full state, the checkpoint, back out to storage so a crash doesn't cost the whole run. Pushing that checkpoint from a large model out to hundreds of GPUs over slow storage turns a single crash into a multi-hour recovery, with the cluster burning power and time while it waits to pick back up where it left off.

Most organizations are trying to solve this by retrofitting storage systems they already own. If teams specify storage requirements before they build the pipeline, rather than discovering the gaps after GPUs start idling, they run into these bottlenecks far less often.

The architectural decisions that separate high-performance object storage from generic alternatives

A handful of specific engineering choices, not the tier name on a pricing page, decide whether object storage can keep a training pipeline fed at GPU speed. Each one below is independent: skip any single one, and the pipeline slows down in its own particular way, regardless of how well the others are handled.

Sustained throughput under concurrency is the first. AI training moves data in large, mostly sequential chunks, but dozens or hundreds of workers are pulling from the same system at the same time. A storage system can post great numbers on a single-stream benchmark and still fall apart once many GPUs hit it simultaneously, so it looks good in a sales deck and bad in production, because most teams overspend on compute and underplan for storage until GPUs start starving for data mid-training. Some teams sidestep the data-movement problem earlier in the pipeline by keeping training data in cloud storage and mounting it directly, using tools like Archil, a serverless filesystem that mounts S3 and GCS buckets as a native filesystem so workloads can read data zero-copy without migrating it first. Parallel file systems such as Lustre and GPFS, along with NVMe-based distributed systems, became the standard for this tier because they're built for heavy concurrency and sequential I/O at the same time.

Cache placement is the second factor. Where the cache sits matters as much as whether one exists at all. If a cache sits somewhere else on the network, data still has to travel before it reaches the GPU, and that round trip costs time even when the cache itself is fast. A cache that lives on or next to the compute node skips that trip entirely. A kitchen in the dining room and a kitchen three blocks away can both cook the same meal, but only one gets it to the table hot.

GPU-direct and RDMA data paths are the third. Technologies like NVIDIA's cuObject and NIXL let data move straight from storage into GPU memory without routing through the CPU first. Skipping that CPU hop removes a bottleneck that no amount of tuning elsewhere in the stack can fully claw back. RDMA for S3 brings the same idea to object storage specifically: data moves from the storage network directly into GPU memory without a CPU copy in between.

Metadata handling and small-object performance make up the fourth. Training and analytics pipelines are full of small files and mixed workloads, and a system tuned only for big sequential reads bogs down badly when it meets that pattern. Metadata tags that support filtering on the server side, by resolution, timestamp, or label, let the storage system hand over only the objects that match, instead of shipping everything across the network and sorting it out later.

S3 compatibility matters, but it's the entry fee, not the differentiator. Throughput, caching, and data paths underneath that protocol are where the real separation happens.

Cost model and elasticity round out the list. High-performance parallel storage billed at a fixed monthly rate for provisioned capacity, SSD-backed Lustre at list price being one example, can end up costing more per month than the GPU compute it's feeding. That flat-rate model made sense for traditional HPC clusters with steady workloads. AI training jobs are bursty, with working sets that swing by orders of magnitude from one job to the next, and storage billed on active use rather than reserved capacity fits that pattern far better.

How the major providers have addressed these architectural requirements

Every provider worth comparing here has gone after one or more of the bottlenecks above, so the useful comparison is on those specific points rather than on marketing language.

DDN's Data Intelligence Platform pairs EXAScaler, a parallel file system built on Lustre, with Infinia, a software-defined object storage system with multi-protocol support. EXAScaler is built for multiple terabytes per second of throughput for large-scale training and checkpointing, and it's running underneath xAI's Colossus cluster, at a scale of hundreds of thousands of GPUs. Salesforce, running on Google Cloud Managed Lustre powered by DDN EXAScaler, saw faster model training, a substantial cut in I/O latency, and lower training costs. Sony Honda Mobility scaled training for its AFEELA Intelligent Drive system three times faster than other Google Cloud options it tried. DDN's strength is raw parallel throughput at HPC scale, and it typically asks for dedicated infrastructure investment in return.

At Google Cloud Next '26, Google announced Cloud Storage Rapid, consisting of Rapid Bucket, a high-performance zonal object store, and Rapid Cache, a read accelerator that bolts onto existing buckets with no code changes required. Google reported meaningfully less blocked GPU time, faster data loading on multi-modal training jobs, and substantially quicker checkpoint restores and writes compared to standard object storage. Google Cloud Managed Lustre has also picked up a substantial bandwidth increase over the prior year's version, with capacity now built out for petabyte-scale workloads. The appeal here is tight integration with the rest of Google's AI infrastructure and zero migration effort for Rapid Cache, but the zonal design means data residency follows Google's own geographic footprint.

CoreWeave's approach centers on its Local Object Transport Accelerator, LOTA, which runs as a proxy service directly on GPU cluster nodes. It acts as a local S3 endpoint and builds a cache out of the disks already sitting on those nodes, so the caching layer is baked into the storage service itself instead of something a customer has to build and babysit separately. LOTA stages frequently accessed objects close to compute so GPUs stay fed through training and fine-tuning, with throughput up to 7 GB/s per GPU and no egress or ingress fees. That's the cache-placement principle from the architecture section, implemented directly: the strength is compute and cache sitting in the same place, with the tradeoff being a tight coupling to CoreWeave's own platform.

NetApp's StorageGRID 12.1 posts meaningfully higher throughput in tested configurations, with the biggest gains appearing in small-object and mixed workloads, exactly the pattern that trips up systems built only for large sequential reads. It also adds bucket change tracking aimed at AI agents, so teams can see precisely which objects changed and skip reprocessing data that hasn't moved, and that saves cost directly in inference and retrieval work, including RAG pipelines.

Dell's AI Data Platform, expanded at NVIDIA GTC 2026, bundles high-performance storage, modular data engines, and a Data Orchestration Engine into one stack. It includes the Lightning File System, a parallel file system rated at up to 150 GB/s per rack, plus Dell Exascale Storage, a single system handling file, object, block, and parallel file protocols on PowerEdge hardware. The pitch is breadth: one platform covering several protocols at once, aimed at on-premises deployments tied into the NVIDIA ecosystem.

Cloudian's HyperStore takes the RDMA path directly, with an NVIDIA RDMA for S3 integration that moves data straight from storage to GPUs and cuts CPU utilization on GPU servers along the way, with throughput over 200 GB/s. It's aimed at on-premises deployments where avoiding public cloud egress fees and packing high storage density onto commodity hardware are the priorities. Reads lean on all-flash NVMe configurations plus RDMA acceleration, and the File Services appliance's read cache uses HDDs while NVMe drives are reserved for write cache and metadata, a version of the same co-location idea that makes CoreWeave's LOTA work, applied here across any cloud or storage backend a customer chooses.

Each approach built hardest where it mattered most for its own customers, including GPU-direct and RDMA data paths. The right fit depends on which specific bottleneck is actually stalling a given team's GPUs, not on which name shows up most often in conference keynotes.