Eliminating Data Download Steps Before GPU Training

Your GPUs may not be the true bottleneck anymore. The wait for data to land on disk before training even starts is what's eating your budget, and most teams don't realize it until they check the utilization dashboard and see a number that makes them wince.

A 2024/2025 arXiv study on I/O performance in ML training pipelines found that poor I/O setups leave GPUs averaging just 45% utilization. Fix the I/O and that number jumps to 95%. Healthy training should sit somewhere in the 85-95% range during active phases; drop below 70% consistently and storage I/O is almost always the reason why. That gap between 45% and 95% isn't a rounding error. Half your GPU fleet sits idle while the meter keeps running.

This is what happens when you split compute and storage across a network without designing for the traffic that crosses it. Every idle GPU cycle on rented hardware is money spent on nothing, and that's the frame worth holding onto for the rest of this piece.

How the download-before-training pattern creates this gap

The old playbook goes like this: copy the whole dataset to local or cluster-attached storage, confirm the copy landed, then kick off training. No step of the actual training runs until the download finishes. Sounds fine on paper. In practice, it breaks in a handful of predictable ways.

Startup latency is the most obvious one. For multi-terabyte datasets, the queue before your first training step even fires can run hours, sometimes days. Then there's storage sprawl: the same dataset gets copied across nodes over and over, and you're paying for disk space that does nothing except sit there being redundant.

Resume overhead stings the most. When a run crashes, and runs crash, teams often replay the entire data loading process from scratch. Teams have documented burning thousands of GPU-hours this way, just re-loading data that was already loaded once.

Then you hit the hard ceiling: if your dataset is bigger than your disk, downloading it is structurally impossible, not just slow. Training on data larger than your available storage gets blocked before it starts. On top of that, training workloads need two very different I/O patterns at once, billions of small random reads for data loading, and huge sequential writes for checkpointing, and most storage setups are built for one pattern, not both.

Meta's own production numbers make the case better than any theory could: 56% of GPU cycles stalled waiting on training data, at petabyte scale. That's the expected outcome of the architecture doing exactly what it was built to do, not a misconfiguration. Which means the fix isn't a bigger disk or a faster network bolted onto the same pattern. The pattern is the problem.

What the market pressure behind this problem looks like

Global AI infrastructure spending topped $250 billion in 2025, and storage and networking investment is scaling alongside compute, with more than half of organizations still reporting bottlenecks, according to Min.io's 2026 AI storage report. You'd think all that money would have solved the bottleneck by now. It hasn't.

More than half of organizations report data and storage bottlenecks limiting their AI performance, and 57% say their data isn't even AI-ready in the first place. Spending money on storage and having storage that actually works for training are, apparently, two very different line items.

The AI storage market is projected to grow from $36 billion in 2025 to $322 billion by 2035. Object storage is the fastest-growing piece of that pie, at a 19.1% compound annual growth rate, pushed by AI training volumes and a broad move away from on-premises file and block storage.

Here's the disconnect worth sitting with: spending is scaling up while the underlying architectural patterns have been slower to shift. Companies are buying more storage. They are not, for the most part, rethinking how their compute talks to it. That gap is exactly why the approaches below are gaining traction in production, not just in research papers nobody reads twice.

Streaming data directly from object storage during training

The idea is simple enough to explain at a bar: instead of copying the dataset before training starts, stream shards from cloud object storage, S3, GCS, Azure Blob, straight to GPU workers as training runs. The download step just... doesn't happen as its own phase anymore.

MosaicML's StreamingDataset, now folded into Databricks Mosaic AI, is the most widely adopted open-source version of this. It solves a few things at once. Training starts the moment a run is triggered, no waiting around. Mid-epoch resumption happens in seconds instead of replaying the whole data pipeline from scratch, which saves real money in egress fees and idle GPU time. Sample order stays identical no matter how many GPUs or nodes you're running, so a checkpoint trained on 64 GPUs can be debugged on 8 without the results going sideways. Data gets converted into MDS (Mosaic Data Shard) format, built for fast random-access reads, which cuts dataloader overhead.

Hugging Face's streaming-enabled datasets follow the same logic: train straight off a hosted dataset, skip the download and local storage entirely.

Worth being honest about the catch: streaming still costs you egress, roughly equal to one epoch's worth of data movement. The libraries do a decent job avoiding duplicate shard downloads across nodes, but you should still run the math against your own storage and compute bill. And streaming libraries are still pulling data over a network on demand, which means latency to object storage becomes the new thing to watch, especially as models grow and batch frequency climbs.

Eliminating the CPU as the data courier between storage and GPU

Even with streaming turned on, data usually still takes a detour through CPU memory before it reaches the GPU. That's a series of memory copies, each one adding latency and eating CPU cycles that could be doing literally anything else.

NVIDIA's GPUDirect Storage (GDS) removes that detour. A DMA engine moves data straight from local NVMe (or NVMe-oF) into GPU memory, no CPU stopover required. The payoff shows up in the numbers: latency drops from 15 microseconds to under 2. Meta's production deployment saw a 3.8x jump in training speed, with data loading throughput climbing from 50GB/s to 192GB/s. PyTorch jobs that used to burn 35% of their compute time just waiting on data stopped stalling.

Checkpointing is where this gets concrete fast. Writing a 140 GB checkpoint through the standard CPU-staged path takes 4 to 5 minutes each time. Run 1,000 checkpoints over a week-long job and you've burned 67 to 80 hours of idle GPU time, which works out to roughly $2,100 wasted at $4.06 an hour per H100 across an 8-GPU node. GDS shrinks that same checkpoint write to under 40 seconds.

One catch worth stating plainly: cloud object stores like S3 and GCS don't support GDS directly. Data has to land on local NVMe first, and GDS applies to that local read after the fact. To get the full benefit you need on-premise NVMe or NVMe over Fabric. NVMe-oF stretches GDS across a network, and centralized NVMe pools can match local drive speeds over InfiniBand or RoCE with RDMA, though the top-tier setups need RDMA-capable NICs. Getting the full benefit requires RDMA-capable networking throughout.

GDS 2.0 shipped with CUDA 12.3+ and added another 15% throughput bump with native support for H100 and H200. Bottom line: GDS is a serious tool, but it's infrastructure-heavy. It fits teams running dedicated on-premise or co-located clusters. It's less suited to training on rented GPUs in the cloud.

Pre-staging data before a job starts, without manual coordination

Here's another angle on the same problem: instead of a training job discovering mid-launch that it needs data, smart orchestration moves the dataset near compute before the job ever runs, based on usage patterns and what's scheduled to run next.

The GPU still ends up reading from local or nearby fast storage. The difference is the wait gets absorbed ahead of time instead of showing up as startup latency the moment you hit launch. Done well, this kills the manual copy-then-verify ritual that precedes most training runs, automatically bumps frequently accessed data up to faster storage tiers without anyone lifting a finger, and cuts down on mid-training stalls from cold storage or cross-datacenter transfers.

What it doesn't fix: data still has to move somewhere. The movement is just earlier and smarter about it. Teams with enormous datasets, constantly shifting run configurations, or unpredictable access patterns can still hit capacity walls or freshness issues that pre-staging can't paper over. Pre-staging works well as an upgrade layered on top of the old pattern, while streaming and GDS take a different path by replacing it outright.

Why object storage is becoming the persistent data layer for training, not just the archive

Object storage used to play a supporting role. It held the canonical copy of the dataset, which then got downloaded to local disk before compute ever touched it, a deliberate staging step, never a live source.

That's flipped. Streaming pulls straight from object storage. Pre-staging reads from it to fill faster tiers. Even the GDS path usually starts at an object store before the data hits local NVMe. Object storage has structural advantages that make this shift make sense: a flat namespace with API-driven access that scales across petabytes without the overhead of managing a file hierarchy, metadata rich enough to locate specific samples at scale (which matters a lot when you're shuffling billions of records), and the ability to let distributed workers hit it in parallel with no coordination overhead.

S3 compatibility has basically become the common language here. Training frameworks, orchestration tools, storage platforms, they've all converged on S3 semantics no matter who's actually running the storage underneath, which means you can move between cloud providers without rewriting your whole pipeline.

The gap that's left: object storage's API model and its eventual-consistency behavior were never built for workloads that also need POSIX semantics, atomic writes, file locking, mmap, fsync. Distributed training jobs lean on exactly those semantics constantly, especially for checkpointing and coordination between nodes.

Where a POSIX filesystem mounted directly on object storage fits into this picture

So you want object storage's scale and its S3 portability, but your training code, your checkpointing libraries, your data loaders, expects POSIX behavior that object storage doesn't hand over natively. That's the actual tension here.

The usual workaround is to download everything to a local filesystem, train against local disk, then push results back up to object storage, which just reintroduces the exact download step this whole conversation is trying to get rid of.

A cleaner approach: mount the object storage bucket directly as a real POSIX filesystem. Training code reads and writes through normal file paths, atomic rename, flock, mmap, fsync, hard links, all of it, while the data itself still lives in the object store underneath.

This is what Archil does. Point it at an existing S3, GCS, R2, or Azure Blob bucket and it mounts as a POSIX filesystem, no migration, no ETL pipeline, no code rewrite. Reads hit an NVMe cache at sub-millisecond speed; on a cache miss, data pulls from the source bucket and gets cached from there, so the dataset never has to fully land on disk before training can begin. Writes replicate before returning, then flush asynchronously back to the bucket, so the object store stays the actual source of truth. Archil doesn't keep a persistent copy of your data outside your own account, and the bucket remains canonical, meaning access can be revoked anytime without any data loss or migration headache.

There's a less obvious reason the file interface matters too: a filesystem is a familiar, well-understood interface, which counts for something once the thing running your training loop is an agent instead of a person typing commands. And because S3, GCS, R2, and Azure Blob all mount the same way, infrastructure teams don't have to rebuild anything just because they switched cloud providers. Capacity is elastic and billed on what's actually cached, so nobody has to guess at worst-case dataset size months in advance.

What a training pipeline looks like when data movement stops being a separate step

Old way: trigger a run, wait for the dataset to download, verify it landed correctly, start training, deal with resume overhead after any crash, push checkpoints back to object storage, repeat the whole thing next time.

New way: trigger a run, training starts immediately while data streams or caches on first touch, checkpoints write back through the mounted filesystem with full POSIX guarantees, and a crash resumes from the last checkpoint in seconds instead of hours.

What actually disappears from a team's workload: the custom data-movement scripts nobody enjoyed maintaining, the excess local storage provisioned to hold datasets that might get read exactly once, the manual download queues teams had to coordinate around a shared cluster, and the practice of replaying data loading from scratch every time a job dies.

Go back to that arXiv number: moving from a poorly configured I/O setup to an optimized one takes average utilization from 45% to 95%. At $4.06 an hour per H100, run that math across even a modest cluster over a week-long job and the difference stops being a rounding error, it's real money. Build infrastructure around where the data already lives instead of constantly shuttling it toward compute, and GPU utilization becomes the binding constraint again. Which, frankly, is the good problem to have.