Eliminating Data Download Steps Before GPU Training

The cluster is reserved. The meter is running. The GPUs are warm, idle, and burning money, and nothing has happened yet because the data hasn't arrived. That gap between "job submitted" and "first forward pass" is the single biggest hidden cost in large-scale AI training, and it has nothing to do with how fast the chips can compute.

Object storage holds the canonical copy of training data for most teams, and for good reason: it's cheap, it's durable, and it was built for the cloud from day one. The architecture simply routes every read through the CPU before it reaches GPU memory, doing what it was designed to do, just not what modern training jobs need it to do.

The standard workaround treats object storage like a slow tier you drain before the real work starts: copy the entire dataset onto local NVMe, then begin training. That staging phase can eat hours before a single gradient gets computed. Picture paying for a stadium full of seats and then making the entire crowd wait in the parking lot until the concession stands finish restocking. That's the download step. The show can't start until the trucks are unloaded.

This isn't a training-only headache, either. The same stall appears one layer up the stack, in AI agents that depend on fast, continuous access to data mid-reasoning: when the storage or network layer introduces latency, the whole multi-step process locks up. The training-time version of that lockup is just the download wait, wearing a different hat.

GPU Underutilization as a Dollar Problem

An idle GPU is a rented sports car sitting in the driveway. Every hour it's not moving is an hour someone is still paying for the lease, because poor storage throughput doesn't just slow a job down, it leaves expensive compute sitting there doing nothing, which turns a plumbing issue into a line item.

The numbers back this up. Research on I/O performance in machine learning training pipelines found that GPUs average only 45% utilization under poor I/O configuration, compared to consistently near-full utilization when the I/O path is tuned properly. That gap doesn't stay flat across a long run, it compounds, because every batch that arrives late pushes the next one back too.

Checkpointing is where this becomes painfully concrete. A large BF16 model checkpoint runs into the hundreds of gigabytes, and writing one through the standard CPU-staged path takes several minutes. A week-long training job needs many of these checkpoints, and across all of them, the wasted GPU-idle time stacks up into entire days of compute that produced nothing but a save file.

The network side tells the same story from a different angle. A single NFS mount serving a multi-GPU training job saturates at a fraction of what modern accelerators can actually consume, and pulling data from S3 one batch at a time adds real latency to every single batch. Neither path keeps the GPUs fed at the rate they're capable of eating.

The usual counterargument is "just prefetch and pipeline around it," and there's some truth there, but only some. Prefetching hides latency when the next batch is predictable and local disk has room to hold it. It falls apart on large shuffled datasets, on multi-node jobs all sharing one NFS mount, and on any job that needs to restart from a checkpoint sitting somewhere remote. In other words, prefetching is a good patch for a small leak and a bad patch for a burst pipe. What matters isn't whether this inefficiency is annoying but whether it's structurally avoidable.

Streaming from Object Storage vs. Downloading First

It is avoidable, and the fix starts with a mental shift: stop treating object storage as a remote warehouse you have to empty before work begins, and start treating it as a live source the training process reads from directly. Downloading stages a static snapshot onto local disk and trains against that snapshot. Streaming trains against the source itself, pulling data the moment the training loop asks for it, so the dataset never has to fit on local storage and the job never has to wait for a copy operation to finish.

AWS SageMaker Fast File Mode shows what this looks like in production. At job start, it identifies the relevant S3 files but doesn't download them. Training starts right away. The S3 objects show up through a POSIX-compliant filesystem interface, as if the files were already sitting on local disk, streaming their contents on demand as the training script reads them. The dataset no longer has to fit on the training instance at all.

Amazon S3 Files follows the same idea from a different angle: applications treat an S3 bucket like a local filesystem, using plain file operations instead of specialized cloud storage calls. That means a machine learning team can point a training job straight at S3 and start running, with no copy step in between.

None of this works by accident. Training frameworks, data loaders, and checkpoint libraries are all written against filesystem semantics, open, read, seek, write, not S3 API calls. Present object storage as a mountable filesystem, and existing code runs without anyone touching a line of it.

The obvious objection is that streaming adds latency to every batch request that a local copy wouldn't have. That's a fair concern, and it's the subject of the next section. For now, the point stands on its own: the interface layer is what changes here, not just the network speed underneath it.

Bypassing the CPU Bottleneck at the Hardware Level

The actual ceiling on streaming performance has never been the network or the storage system itself; it's the CPU bounce buffer that every read has to pass through before the data reaches GPU memory. Data takes a detour through the CPU like a cross-country flight that connects through an airport nowhere near either city. Recent hardware closes that detour entirely.

NVIDIA cuObject extends the GPUDirect approach to S3-compatible endpoints directly. It's a library suite built for direct transfers between GPU memory and object storage over RDMA, using x-amz-rdma-token headers to push and pull data straight into GPU memory over Dynamically Connected RDMA. The POSIX scratch-staging step is simply gone.

The pace of shipping here has been fast. cuObject v1.0.0 shipped in January 2026 alongside CUDA 13.1.1, and v1.2.0 followed on May 26, 2026 with CUDA 13.3, splitting the control plane (the HTTP S3 requests) from the data plane to reach wire-speed transfer straight from object storage to GPU memory. NIXL plays a supporting role in this picture: it's the abstraction layer that lets different transfer backends, including the object storage path, get selected without the application needing to know which one it's talking to.

The Dell and NVIDIA ObjectScale deployment puts a real number on what this buys. vLLM, LMCache, and NIXL offload KV cache to Dell ObjectScale over RDMA, landing straight in GPU memory via cuObject, no fork, no custom client required. The trade that results: a GPU rebuild of context that used to take multiple seconds gets replaced by 837 milliseconds of storage retrieval, freeing the accelerator to go back to doing the work only it can do.

None of this is free of requirements, and it's worth being straightforward about the catch. cuObject needs RDMA-capable networking, a Mellanox ConnectX-5 or better InfiniBand HCA, or a RoCE-capable Ethernet adapter. Teams without that fabric sitting in their racks can't take the zero-copy path yet. For them, the filesystem-layer approach covered further down is the practical route in the meantime.

What Hyperscalers and Storage Vendors Have Shipped

None of this is a lab demo. Multiple vendors have already shipped the plumbing, and they arrived at nearly the same design independently, which is the strongest evidence that streaming from object storage is the direction the industry is actually heading.

Google Cloud Managed Lustre, announced at Google Cloud Next 2025, delivers high aggregate bandwidth with up to 80 petabytes of capacity, and it can use TPUDirect and RDMA to move data straight to accelerators, bypassing the host entirely. Google's Rapid Buckets adds sub-millisecond latency and very high operations-per-second to Google Cloud Storage, which lets large-scale training checkpoints and recoveries happen almost instantly. Google reports accelerators holding very high utilization as a result.

AWS's contribution comes in two complementary products. SageMaker Fast File Mode lets training start without waiting on any data download, presenting S3 objects as a POSIX-compliant local filesystem and streaming content on demand, with no requirement that the dataset fit on instance storage. Amazon S3 Files lets applications read and write S3 data using standard file operations, so ML teams can run training jobs straight against S3 with no copy step.

NVIDIA's Storage-Next initiative takes a broader, standards-setting approach. It brings together a wide coalition, storage makers, controller vendors, hyperscalers, and application developers, to build open standards for GPU-driven storage, aimed squarely at the kind of small, accelerator-initiated requests that today's fragmented protocols handle poorly. IBM has already demonstrated interoperability between a SCADA client and a prototype Storage Scale server built on the SDK.

Cloudera's Lakehouse takes the most conceptually distinct approach of the group. Rather than speeding up data movement, it removes the need for movement in the first place: feature stores, training datasets, and retrieval corpora all share the same Iceberg tables, using snapshots to freeze a consistent view for training while new data keeps flowing in for inference. Workloads run where the data already sits. It is a structural answer rather than a mechanical one, which raises the next question: what do teams do when they're not running on a hyperscaler's managed service?

A High-Performance Filesystem Layer Over Your Existing Bucket

Managed services like SageMaker Fast File Mode are great if the training stack already lives inside that ecosystem. RDMA fabric and cuObject are great if the racks already have the right network cards installed. But plenty of teams have training code, data loaders, and checkpoint logic that were simply written against standard file system calls, years before any of this was available, and for them, mounting object storage as a real POSIX filesystem is the most direct way to drop the download step without touching a line of code.

The logic is straightforward. Training frameworks and data pipelines are built around open, read, write, seek, the same POSIX calls every developer and every shell script already knows by heart. Present object storage through that same interface, and the training script reads local NVMe and a cached S3 bucket identically.

Full POSIX semantics, atomic rename (used by checkpoint libraries to avoid partial writes), flock and fcntl (used by distributed launchers), mmap (used by memory-mapped dataset formats), fsync, hard links, symlinks, and sparse files all have to actually be there, since if any of these are missing, existing code breaks silently or visibly. Skip any of them, and existing code breaks, sometimes loudly, sometimes silently and worse for it. A filesystem layer that cuts corners on these is a different thing wearing local disk's clothes, not a lighter version of local disk.

NVMe caching at this layer is what closes the latency gap without requiring any RDMA fabric at all. Reads that hit the NVMe cache come back at sub-millisecond latency, and on a cache miss, data gets pulled from the source bucket and cached for next time. The training loop experiences fast local performance on hot data without ever downloading the full dataset up front.

Through all of this, the bucket never stops being the source of truth. The filesystem layer doesn't keep a persistent copy of data outside the customer's account, access can be revoked at any time, and the data never leaves where it already was. No migration, no ETL pipeline to maintain, no lock-in on the data itself.

That property is what makes multi-cloud and hybrid setups workable rather than theoretical. The same mount works against S3, GCS, R2, and Azure Blob with no code changes needed. Training in one cloud against data sitting in another becomes a configuration choice made in an afternoon instead of an engineering project that takes a quarter.

What changes operationally when the download step is gone

Removing the download step does not just shave time off a single run. It changes what a training pipeline looks like day to day.

Cluster reservations shrink to the size of the actual training job, instead of training time plus however many hours it takes to pre-stage a dataset that may be terabytes wide. Local disk no longer has to be sized for the full dataset, since a filesystem layer with NVMe caching only needs room for whatever's hot at the moment, not a complete copy of everything. Multi-cloud and hybrid training stop being special projects that need their own migration plan, because the mount handles that automatically across providers.

Checkpointing stops being a tax paid in GPU-idle hours. Restarts from a remote checkpoint become routine rather than risky, since the data path no longer depends on a slow CPU-staged copy to get moving again. And teams get to keep their existing code, their existing data loaders, their existing checkpoint libraries, because the whole approach is built to look like local disk from the application's point of view.

None of this requires betting the training pipeline on a single vendor's managed service or a specific network card. It requires recognizing that the download step was never a fact of nature. It was a workaround for storage that couldn't talk to GPUs directly, and that workaround now has better options sitting right next to it.