Object Storage Solutions for ML Pipeline Data

Object storage won the job of holding machine learning data because its design matches the shape of the work, beyond simply costing less per gigabyte than the alternatives. A flat namespace and HTTP-based parallel access let it handle the exact pattern that AI training creates: lots of readers, all at once, all pulling different pieces of data with no need to wait their turn. Each object can be reached on its own, with no file locks and no traffic jams, so dozens of nodes can read at the same time without stepping on each other. That is the property that matters when a training job fans out across a GPU cluster: nobody's waiting in line.

The practical glue holding this together is the S3 API, which has turned into a shared language that nearly every tool in the ML stack already speaks. Apache Spark, MLflow, Kubeflow, and Airflow all support S3 out of the box. TensorFlow needs the separate tensorflow-io plugin to get there, and PyTorch relies on the external Amazon S3 Connector for PyTorch library, but either way, a team can point its pipeline at any S3-compatible store and start working without rewriting code to migrate.

That compatibility enables something that older storage setups simply couldn't do: split compute from storage. A data engineering team can load in more training data while the GPU fleet keeps running, and the AI team can add more compute without touching the storage layer at all. If you try that with a traditional SAN or NAS setup, you'll hit a wall, because those systems were built assuming compute and storage grow together, not independently. Object storage broke that assumption. The S3 API didn't just make object storage convenient. It made object storage the backbone that every stage of the ML pipeline, from the moment raw data lands to the moment a model serves predictions, can plug into without translation.

Storage demands across pipeline stages

A machine learning pipeline is five or six separate jobs, each with its own I/O needs, stitched together, and a storage layer that handles one stage well but another poorly will choke the whole chain.

Raw ingest comes first, and it's a bandwidth fight more than a speed fight: huge volumes of semi-structured and unstructured data pouring in, with no fixed schema, often reaching petabyte scale. The requirement is write throughput and the ability to accept whatever format shows up, with fast response times not required. Object storage handles this cleanly because it doesn't care what shape the data is in, and it plugs directly into ingestion tools like Airflow, Kafka, and Prefect.

Next comes data preparation and cleaning, where things get more sensitive to delay. Spark and dbt can read straight from object storage over S3, but repeated iterative jobs sometimes need a faster tier sitting in front of it. The bottleneck here often has nothing to do with throughput. When a job has to scan millions of Parquet files, the slow part is listing those objects, not moving the bytes.

Training is where the real stress test happens. Stochastic Gradient Descent shuffles the training set randomly on every single epoch, which turns into hundreds of thousands of parallel, random reads scattered across a massive dataset. This is the stage where plain object storage tends to break down: Meta found that 56% of GPU cycles sat stalled waiting on training data, a number that shows how badly an HTTP round-trip penalizes this kind of access pattern at scale.

Checkpointing runs in parallel with training but flips the I/O direction. Models have to dump their full state to storage fast, so a failed node doesn't wipe out hours or days of progress. PyTorch's standard way of writing these checkpoints atomically has a history of clashing with how S3 handles writes, and that's a real engineering headache teams run into, not a hypothetical one.

Fine-tuning works with smaller datasets than pretraining, but the access pattern turns more iterative, and latency becomes a factor again. Inference and serving flip the profile once more: object stores are picking up traction here too, but the demands shift toward high concurrency and fast single-object retrieval rather than sustained throughput. Archiving closes the loop: cold data, cost is the only variable that matters, and lifecycle rules handle it without anyone touching it.

GPU starvation's cost and storage's role in it

When storage can't keep up with how fast a GPU cluster wants to read data, the GPUs sit idle, and idle GPU cycles turn into the single most expensive line item on the infrastructure bill. Storage is the reason that line item exists.

Training runs that hit a data-loading bottleneck often see GPU utilization fall sharply, while the same cluster with properly tuned I/O can run at a consistently high utilization rate. The space between those two numbers is pure waste, and it's waste measured in the cost of the most expensive hardware most companies own. Blackwell-class GPU nodes carry enormous capital costs and power costs every single year they're in service, so every idle cycle is money leaving the building.

Stability AI's experience makes the case concretely. Switching to a higher-performance storage setup, without touching the model or the compute cluster at all, pushed GPU utilization up dramatically. Nothing about the model changed. Nothing about the GPUs changed. The only variable was how fast data reached them, and that one variable moved utilization by a wide margin.

Think of a GPU cluster like a kitchen full of chefs who can each plate a dish every few seconds, as long as ingredients keep arriving. Slowing the delivery truck down means it doesn't matter how fast the chefs can cook. They stand around with empty cutting boards, and the restaurant still pays their wages while they wait. That's what storage-driven GPU starvation looks like on a budget sheet: full staff, no output, same cost. The fix is a storage decision made before the first training run starts, which the next section takes on.

The tiered storage architecture that keeps GPUs fed through training

You don't fix training's relentless demand for random reads by walking away from object storage. It's putting an NVMe cache layer in front of it, one that can absorb the latency object storage can't deliver on its own. Object storage still wins on throughput for large files and sequential transfers, but block storage wins on IOPS and does worse at bulk transfers, so you use each where its strengths apply, not one system forced to do every job.

Production deployments tend to converge on the same three-tier pattern, moving from a cold raw tier, to a hot active-training tier, to a processed curated tier. At the bottom sits the cold, raw tier: object storage holding the source of truth, built for cheap capacity, parallel ingest, and accepting any data format without complaint. Above that sits the hot, active-training tier, where an NVMe cache fronts the object store and delivers sub-millisecond reads that can absorb the scattershot access pattern SGD shuffling creates. At the top sits the processed, curated tier, usually a feature store, read by both the training pipeline and the inference layer so the two don't end up working from mismatched versions of the same data.

GPUDirect Storage adds another piece to this picture: data can move straight from storage into GPU memory, skipping the CPU bounce buffer that used to sit in the middle of every transfer. It's a production-ready option now, and it changes the latency math meaningfully for the hottest part of the training path. Meta's shift away from CPU-bottlenecked loading toward GPUDirect streaming shows what the older architecture was capping out at, and why pairing a cache layer with a direct storage-to-GPU path raises that ceiling.

None of this works if the data itself is laid out badly. Columnar formats like Parquet, sensible partitioning, and files sized appropriately all affect how efficiently a system can read that data, and poor layout can slow reads even on fast storage. A poorly partitioned dataset can bring even the fastest storage to a crawl, while a well-organized one lets a cheaper setup perform just fine. Storage tier and data layout are the same problem looked at from two angles.

What this tier actually looks like in a working pipeline is a cloud filesystem that mounts straight onto an existing S3, GCS, R2, or Azure Blob bucket, with no migration step, no ETL job, and no code changes required. That approach delivers the NVMe-cached, POSIX-accessible hot tier directly on top of the bucket a team is already using as its source of truth. A team can adopt the tiered architecture described above in stages rather than ripping out existing infrastructure and starting over.

Hidden pipeline costs from the file-object split

Running object storage for training while keeping a separate file storage system for everything else can sound like a reasonable division of labor, until someone has to keep both copies of the dataset in sync. That sync job is a hidden tax, and it grows heavier as the pipeline grows.

ML pipelines routinely mix file-based ingestion over NFS or POSIX with object storage accessed through the S3 API, especially for staging large datasets. Each translation step is a small tax on its own, but pipelines run these translations constantly, and the small tax adds up into a real drag on throughput.

A workflow tool like DVC illustrates how deep this runs. DVC stores data in a configurable backend, which could be S3, Google Cloud Storage, Azure Blob Storage, or a plain filesystem, but its metadata layer still expects filesystem semantics underneath it. That expectation doesn't disappear just because the data happens to live in an object store.

The stronger fix is a layer that gives a bucket full POSIX semantics: atomic rename, flock and fcntl, mmap, fsync, hard links, symlinks, the whole toolkit. With that layer in place, existing code runs as written, with no SDK rewrite and no rearchitecting, and the object bucket stays the single source of truth the whole time. Instead of maintaining two copies of a dataset and a sync job to keep them honest, a team gets one copy of the data that behaves like a filesystem when something needs a filesystem, and an object store when something needs an object store.

Egress costs and data residency as structural risks in cloud object storage choices

Getting the architecture right doesn't guarantee the bill stays reasonable. Egress pricing can turn a well-built object storage setup into a surprisingly expensive one, and the gap between providers is wide enough to change which provider makes sense in the first place.

If training data sits in one region and the GPU cluster runs in another, every single training run pays an egress fee on every byte read. AWS charges roughly $0.09 per gigabyte for data leaving the platform, and moving large training corpora across regions at that rate adds up fast. Google Cloud Storage runs the highest egress rate among the big three cloud providers, and pulling a large corpus out at that rate turns into a cost that recurs with every major training run and every data export.

A handful of providers built their pricing to avoid this problem. Cloudflare R2 charges no egress fees at all, with no caps and no fine print about "reasonable use," making it a release valve for teams whose workloads involve predictable, large-volume reads. OVHcloud dropped its egress fees in January 2026, making it a zero-egress option for EU teams with GDPR compliance requirements. On the on-premises side, Cloudian HyperStore runs S3-compatible object storage locally and presents a single namespace across every cloud a company uses, with the same S3 APIs throughout, so it sidesteps cross-region egress by keeping the data where the compute is while it still speaks the S3 language everything else expects.

Teams in regulated industries need cache nodes sitting next to their compute, and no data can cross a border unless someone explicitly configured it that way. Residency has to be a choice the customer makes on purpose, rather than a default a vendor sets quietly in the background.

One upside of the S3-compatible ecosystem: switching providers usually means changing an endpoint URL and a set of credentials, not rewriting application code. That lowers the switching cost considerably. What it doesn't erase is the cost of physically moving an existing dataset to a new provider in the first place, since that migration itself runs straight through the same egress fees described above. Choosing a provider well the first time avoids paying the cost of choosing poorly later.