High-performance object storage providers for AI data pipelines
Storage bottlenecks cut GPU utilization in half, and most cloud providers can't fix it.
AI infrastructure spending jumped roughly 166% in 2025, almost all of it aimed at GPUs, while the plumbing that feeds those GPUs got a fraction of the attention. Cloud object storage grew from $8.14 billion in 2024 to $9.49 billion in 2025, a 16% jump, and object storage is outpacing every other storage type at 19.1%. That growth isn't archived files sitting untouched. It's training data, checkpoints, and inference traffic, and 57% of enterprises still say their data isn't ready for any of it. Buying GPUs used to be the hard part, but now the harder question is whether your data can even reach them in time to matter.
What GPU underutilization actually costs when storage is the bottleneck
Bad I/O setup gets you around 45% GPU utilization, while fixing it lands you near 95%. That 50-point gap is the entire cost of picking the wrong storage setup, and it shows up quietly in your monthly cloud bill like a fee you didn't notice signing up for.
Most people in this field treat 85% utilization as the floor for decent ROI on accelerator spend. Fall below that and training runs stretch out for weeks while expensive chips sit idle. A single idle Blackwell-class node burns roughly $30,000 a year in wasted power and capital, a truck payment gone for hardware that spent the year staring at nothing.
Meta ran into this at scale, and Meta doesn't exactly run sloppy infrastructure. Even with exabytes stored in a distributed filesystem built for this exact job, 56% of their GPU cycles stalled out waiting on training data to show up. They had to build an entirely separate system, called Data PreProcessing Service, just to keep GPUs fed on schedule. If Meta's engineers hit this wall with a huge budget and some of the best infrastructure talent around, a mid-sized team running off-the-shelf storage and good intentions isn't going to fare better.
A financial services company training a fraud detection model found this out the expensive way. They spent millions on accelerators, and the cluster still sat at very low utilization. Nobody could blame the model this time, or the code, or bad luck. The real problem was a storage setup delivering 8 GB/s of sustained throughput to workloads that needed several times that. Storage acts as a multiplier on everything else in the stack, and get it wrong, every dollar you spend on GPUs comes back worth less than a dollar.
What object storage must actually deliver at each stage of an AI pipeline
Object storage used to be where files went to retire, a cold tier nobody thought much about. Now it's load-bearing at nearly every step of an AI pipeline, and each step wants something different from it. That's exactly why "fast storage" means nothing on its own; fast for what?
Raw ingest has to swallow petabytes of images, video, sensor logs, and text arriving from a dozen sources at once, without choking on parallel writes. Training wants something else entirely. It's throughput-bound, and large language models can pull 10+ GB/s of sustained reads, with computer vision workloads often asking for more. A peak bandwidth number on a spec sheet tells you almost nothing here; what matters is how many readers the system can serve at once before it buckles. And no amount of fast hardware saves a badly organized dataset. Columnar formats like Parquet, sane partitioning, right-sized files, these matter as much as the hardware underneath. Fast storage grinding through poorly partitioned data behaves like a race car stuck behind a school bus on a two-lane road.
Checkpointing brings its own headache. Multi-terabyte writes, dumped simultaneously from thousands of GPUs, need somewhere to land fast. Do it synchronously and training stalls completely until the write finishes, an outage that can cost millions in idle GPU time over a long run. Inference flips the problem: it needs low-latency reads, often thousands of sessions hitting the same objects at once, on a tight clock.
No single provider handles all four of these well. That's the whole reason picking a storage vendor isn't a checklist exercise. It's a real decision with real consequences attached to it.
How data movement architecture separates capable providers from fast-looking ones
The path data normally takes to reach a GPU is almost comically roundabout: storage, kernel buffer, userspace, driver, GPU memory. Every hop adds latency and burns CPU cycles that should be doing something more useful with their time.
GPUDirect Storage (GDS) cuts that chain out entirely, dropping latency from roughly 15 microseconds to under 2. Meta saw a 3.8x jump in training speed after rolling GDS out across its research clusters, just by removing the CPU as a middleman for data loading. GDS 2.0, shipping with CUDA 12.3 and up, adds another 15% throughput on top of that, with native support for H100 and H200 chips.
Here's the part nobody puts on the slide deck: cloud object stores, including S3 and GCS, don't support GDS directly. Data has to land on local NVMe first before GDS does anything useful, so the speed boost only covers that last local hop, not the download from the cloud itself. A plain cloud bucket, whatever the marketing says, can't be the whole answer for high-throughput training, and that's a structural limit, not a tuning problem. Something has to sit between the bucket and the GPU, either a caching layer or an on-premises system, or the GDS advantage never shows up at all.
NVMe-over-Fabrics setups get around part of this by letting bandwidth scale independently of raw capacity. Teams moving off older TCP-based storage arrays onto purpose-built NVMe-oF fabrics report GPU utilization gains of 70 to 80%. So the real question for any vendor isn't the speed number on their spec sheet, it's where the data actually stalls on its way to the chip.
How the leading high-performance object storage providers approach AI pipeline fit
MinIO AIStor is S3-compatible, built for AI training, and runs on-premises or in a private cloud. It claims roughly 5x throughput gains for training and inference using GPUDirect RDMA, skipping the CPU in the data path entirely. Good fit if you want open-source roots and full control over your environment. The catch: it needs dedicated infrastructure, so pointing it at your existing cloud bucket and walking away isn't an option.
VAST Data runs an all-flash, disaggregated setup with one namespace covering object, file, and database access. Its native asynchronous checkpointing solves the stalled-training problem directly, and it handles mixed workloads (training, inference, analytics) hitting the same data without much drama. It's mostly an on-premises or co-location play though, which makes it a tougher sell for teams already committed to public cloud object storage.
WEKA pairs a parallel file system with an S3-compatible interface. Its WEKApod reference setup hits 720 GB/s of throughput from just 8 storage nodes, feeding 768 H100 GPUs. Numbers that sound made up until you check the spec sheet twice, honestly. It's built for the training tier specifically, the high-concurrency, metadata-heavy random reads that bring traditional object stores to their knees. It tiers out to cloud storage for cold data fine, but the fast layer still needs dedicated WEKA hardware sitting underneath it.
VDURA announced general availability of RDMA support and something called Context-Aware Tiering at GTC 2026. The tiering feature moves data across storage tiers on its own, cutting down on manual babysitting, and RDMA lets GPU servers reach storage directly over the network without looping the CPU in. It's newer to general availability than the others on this list, so the production track record is thinner.
Wasabi and similar cloud-native S3-compatible providers offer a cheap, reliable option for raw ingest and cold dataset storage, and S3 API compatibility keeps integration friction low. They fit better as the data lake layer in a bigger setup than as the hot store your GPUs read from directly; they weren't built for training-tier throughput. And the same GDS limitation applies here too: no direct GPU data path unless you put a local NVMe cache in front of it.
Archil takes a different angle entirely. It mounts your existing S3, GCS, R2, or Azure Blob bucket as a POSIX filesystem, no migration, no ETL pipeline, no code changes. An NVMe cache sits in front of the bucket, so reads that hit cache come back in under a millisecond; misses just fetch from the source bucket and fill the cache for next time. That closes the GDS gap from the last section. By giving GPUs a local NVMe-backed filesystem to read from, Archil lets you use GDS without ripping out cloud storage you already run. Writes get replicated before the call returns, then flushed to the bucket asynchronously, so checkpointing doesn't force training to sit around waiting. It supports full POSIX behavior, atomic rename, file locking, mmap, fsync, hard links, sparse files, so training code runs as-is instead of getting rewritten around a vendor's quirks. Capacity gets billed on what's actively cached, not some fixed tier you guessed the size of six months ago. The bucket stays the source of truth the whole time; Archil never holds a lasting copy of your data outside your own account. It's the one option on this list built to add speed on top of a bucket you already own, instead of asking you to move house.
The tiered architecture that most production AI pipelines are converging on
Nobody runs one storage tier for everything anymore, because no single tier does every job well. What's showing up instead is a layered setup, matched tier by tier to what each pipeline stage actually needs.
The raw data lake sits at the bottom: a cost-optimized, S3-compatible store like Wasabi, plain S3, or GCS, where capacity and API compatibility matter more than raw speed. Above that sits the hot training tier, a high-throughput parallel store or an NVMe-cached filesystem, and this is exactly where WEKA, VAST, and Archil (used as a cache layer over the lake) end up competing head to head. The checkpoint target needs to absorb bursts and flush asynchronously, so async write support is the one line item worth checking before you assume it's handled. The inference path wants fast, low-latency reads at high concurrency, usually served off that same hot tier or a dedicated cache sitting beside it.
On the hardware side, PCIe Gen5 NVMe drives pushing past 14 GB/s sequential reads are becoming the default for that hot tier. Feature stores paired with tiered caching are cutting cold-data access latency by 10x in some early rollouts. None of that matters much, though, if the data underneath is a mess. File format choices like Parquet, partitioning strategy, and file sizing affect read speed at every tier, hardware or not. Teams that spend big on a fast hot-tier store while ignoring how data moves between tiers have solved maybe half the problem, and they usually find out which half the hard way.
The evaluation criteria that separate providers by AI pipeline fit, not just marketing claims
Vendor benchmarks love showing off peak throughput. Ask instead for sustained throughput under real training-scale concurrency, ideally benchmarked against plain S3, EFS, or EBS defaults, something you'd actually recognize from your own bill.
Ask about the data path itself too. Does the provider support a direct route to GPU memory through GDS or RDMA, or is data still bouncing through CPU memory on every single read? If it's cloud-based, is there a local NVMe layer that makes GDS possible on that last hop, or is that hop just missing entirely?
Migration cost matters more than most teams admit going in. A provider that demands you move your data or adopt a new S3 endpoint adds real risk and real delay to a project that's usually already behind schedule. A layer that mounts on top of buckets you already have, with zero code changes, costs a lot less to adopt, in both time and headaches.
Find out whether checkpointing runs synchronous or asynchronous, since that one detail decides whether your training run pauses every single time it saves progress. Confirm POSIX support while you're at it, since data loaders lean on atomic rename, mmap, or file locking behind the scenes more than people realize, and a provider missing these forces you to rewrite code you never planned on touching.
Look at how capacity gets billed too. AI workloads swing wildly in size, and a fixed, pre-provisioned tier either wastes money during a small experiment or runs out of room during a big one; metered, active-use billing fits the actual shape of the work better. And check where the data lives at rest. The vendor should never hold a lasting copy of your training data outside your own account, and walking away from the contract should leave the data sitting exactly where it started.
A provider earns its spot in the pipeline by cutting unnecessary data movement, keeping GPUs fed without stalling, and fitting into what you've already built. One that asks you to rebuild your pipeline around it is asking a lot in exchange for a promise of speed, and that's a trade most teams end up regretting somewhere around month three.
