Object Storage Solutions for ML Pipeline Data

ML pipelines have six distinct stages, and each one asks something different from storage. Most evaluations skip this part entirely. That's why teams end up with a setup that works great for two stages and quietly destroys performance for the other four — like a car with a Ferrari engine bolted to a bicycle frame.

Here's what each stage actually needs:

Raw ingest. Semi-structured and unstructured data (images, video, sensor outputs, raw text) arrives at petabyte volumes. Throughput and capacity are what matter. Latency is secondary because nothing is waiting on this data in real time.

Data preparation and preprocessing. Tools like Apache Spark read chunks sequentially, process them in memory, and write results back. Bulk sequential throughput is what you want. A Microsoft AI Infrastructure Architect cited by Blocks & Files put it plainly: cheap object storage plus a CPU cluster beats an expensive file system here. The bottleneck is almost always pipeline design, not storage speed.

Training. This is the stage that breaks most storage setups. GPU clusters need a continuous, high-throughput data stream. If storage can't keep pace, GPUs stall. The requirement isn't just high throughput in a benchmark. It's sustained throughput under the real read pattern of a distributed training job, which is a harder thing to actually deliver.

Checkpointing. Periodic full-state saves of model weights, often hundreds of gigabytes at a time. This one gets treated as an afterthought constantly, and it shouldn't be. Synchronous checkpoint writes stop every GPU in the cluster until the write finishes. Whether that pause lasts ten seconds or ten minutes comes down entirely to how the storage layer handles it.

Serving and inference. Lower sustained throughput than training, but latency matters more here. Model weights have to load fast at serve time. A slow first load means a slow cold start, and in production inference, that shows up immediately in p99 latency numbers.

Archival and compliance. Older datasets, previous model versions, prompt logs. Object storage is the natural home for all of it. The requirements are durability, low cost, and the ability to pull data back for future retraining. Speed is not the point.

No single tier handles all six stages well. The real architecture question is which stages need a dedicated performance layer and which can stay on plain object storage. Checkpointing deserves special attention because it hits GPU utilization on a recurring, predictable schedule. Most teams find that out the hard way.

GPU Idle Time Is the Real Cost of Treating Object Storage as a Passive Data Lake

A 2025 Run:ai analysis found that nearly 40% of enterprise GPU idle time comes from I/O wait. That's the GPU sitting there, fully powered, fully leased, doing nothing while storage tries to catch up. In unoptimized setups, data preprocessing alone can eat up to 65% of epoch time, and GPU utilization can settle in the 60–70% range. Not because you bought the wrong GPU. Because your data delivery can't keep up with it.

Meta ran into this at frontier scale. 56% of GPU cycles were stalling while waiting for training data. The fix required building an entire custom Data PreProcessing Service just to unblock the pipeline. Most teams are not Meta, which means they're dealing with the same problem without a team of engineers to build something custom around it.

The financial math isn't complicated. Blackwell-class GPU nodes run roughly $30,000 per node per year in capital and power costs. When storage can't keep them fed, that money is paying for idle hardware. It's like renting a racetrack and spending most of the day in the pit lane.

The physics are pretty simple too. Each modern GPU can consume 1–5 GB/s during training. An 8-GPU node needs 10–40 GB/s in aggregate. Standard S3-style endpoints weren't built to deliver that. They were built to store things cheaply and retrieve them occasionally. Delivering a sustained multi-gigabyte-per-second stream to a distributed GPU cluster is a different job entirely.

Checkpointing makes it worse on a schedule. For large models, each checkpoint write can run to hundreds of gigabytes. Synchronous writes pause the whole cluster. The cost isn't random. It shows up at every checkpoint interval, like a tax on a fixed timer.

The data lake stores your data safely. Active training infrastructure delivers it fast enough that your GPUs never have to wait. That's the actual difference between the two.

The Capabilities That Separate Passive Object Storage from Active ML Infrastructure

When you're evaluating object storage for an ML pipeline, the things that matter aren't the ones on a standard spec sheet. Here's what to actually look for:

Throughput at GPU scale. Can the storage layer saturate a multi-GPU cluster under real training conditions, not just in a single-client benchmark? MLCommons published the MLPerf Storage v2.0 benchmark in August 2025, which now includes checkpointing tests and shows tested systems serving roughly twice the accelerator counts of v1.0. It's a useful neutral reference when vendors start making throughput claims.

Filesystem semantics without data movement. Does your team have to copy data out of the bucket before training starts, or does the storage layer present the bucket as a mountable filesystem? PyTorch and JAX assume filesystem-like access: atomic rename, fsync, flock, mmap, hard links. If a solution requires wrapping every read in an SDK call, that adds latency and means rewriting training code. Neither is acceptable.

Caching architecture. An NVMe cache layer between the bucket and the GPU cluster turns object storage latency into sub-millisecond reads on hot data. Ask whether the cache is persistent across runs, whether it scales up and down with the workload, and whether it warms on first access or requires someone to manually prime it before a job starts.

Checkpointing behavior. Asynchronous checkpoint writes that flush to the bucket without pausing the cluster are the dividing line. Solutions that do this treat checkpointing as a feature. Solutions that don't make you pay for it in GPU idle time on a predictable schedule.

Elasticity and billing model. Training jobs vary a lot in working-set size. A solution that charges for provisioned capacity regardless of what's actively cached punishes the variable nature of ML workloads. Billing on actively cached data fits the actual usage pattern much better.

Cloud portability. Buckets live on S3, GCS, Azure Blob, R2, and sometimes all four at once. A storage layer that mounts them all the same way means no migration when compute moves. Lock-in at the storage layer is expensive to undo.

Data residency and access control. The bucket should stay the source of truth at all times. If you revoke the storage layer's access, your data should be exactly where it was, with no persistent copies held outside your account.

How the Major Object Storage Solutions Compare on ML Pipeline Fit

The comparison below focuses on what matters for ML pipelines: throughput, filesystem interface, caching, checkpointing, cloud portability, and pricing model.

Amazon S3 (with S3 Express One Zone)

S3 is native object storage. There are no POSIX semantics without a translation layer. Mountpoint for S3 gives you read-heavy POSIX access but with real limitations, particularly for write-intensive or mixed workloads. S3 Express One Zone cuts per-request latency meaningfully but stays single-AZ and carries a pricing premium.

For archival and ingest, S3 is a sensible default. For active training, it typically needs a data-copy pipeline sitting in front of it. And getting good throughput per bucket requires careful partitioning to avoid hitting bottlenecks at GPU scale.

MinIO (AIStor)

S3-compatible, deployable on-premises or in the cloud. MinIO's throughput story for self-managed deployments is strong, and the vendor publishes numbers at a scale that would saturate large GPU clusters. Independent replication of those figures at comparable hardware configurations is limited, so take vendor benchmarks as a starting point rather than a final answer.

The tradeoff is operational: you're running and scaling the storage cluster yourself. Strong fit for teams that want on-prem control and have the engineering time to operate infrastructure.

Cloudflare R2

S3-compatible with zero egress fees. That's a real cost advantage when training clusters pull the same data repeatedly across sessions. For ingest and archival, R2 is worth a close look. For active training at GPU scale, throughput and POSIX semantics still need help from additional tooling.

Google Cloud Storage (GCS)

GCS integrates natively with Vertex AI and TPU clusters, and performance is strong when compute and storage sit in the same GCP region. Cross-cloud or hybrid setups require more work to make function. If you're all-in on GCP, it's a natural fit. If you're not, the advantages narrow fast.

Archil

Archil mounts existing S3, GCS, R2, or Azure Blob buckets as POSIX filesystems. No migration. No ETL step. No changes to training scripts.

On reads, hot data sits in an NVMe cache layer and serves at sub-millisecond latency. Cache miss means the data gets fetched from the source bucket and cached on the way through. On writes, data replicates before returning, then flushes to the bucket without blocking the training run. The bucket stays the source of truth, and no persistent copy sits outside the customer's account.

Capacity is elastic and billed on actively cached data, not provisioned volume, which fits the variable working-set sizes of ML training jobs. There's also a serverless execution layer that lets agents or training jobs run commands against the filesystem without a separate sandbox, which matters for preprocessing and inference pipelines that need to manipulate files directly.

Because Archil is cloud-agnostic, S3, GCS, R2, and Azure Blob all mount the same way. Teams aren't locked to any single cloud's native storage performance.

The fit is specifically for teams who want to run training directly against an existing bucket at filesystem speeds, without building a data-movement pipeline to get there.

Quick reference

• Archival and ingest at lowest cost: S3 or R2
• Self-managed high-throughput on-prem: MinIO
• GCP-native TPU workloads: GCS
• Running training directly against any existing bucket at filesystem speeds, no data movement: Archil

What "No Data Movement" Means in Practice for a Training or Inference Team

The standard ML storage workflow goes like this. Data lands in a bucket. An engineer writes a pipeline to copy it to a local NVMe volume or EFS share. The training job reads from the copy. The copy has to be refreshed whenever the source data changes. Then it starts over.

That pipeline costs more than it looks like on paper. There's the engineering time to build and maintain it. There's the storage bill for the duplicate copy. There's the lag between source updates and training-ready data. And there's the coordination overhead when multiple teams need the same dataset, possibly in different formats, at the same time. It adds up fast — a data-movement tax that compounds with every new project.

A 2025 AI storage architecture report from MinIO found that 57% of enterprises say their data isn't AI-ready even as experimentation accelerates. A big share of that gap isn't about the data itself. It's the movement and preparation burden sitting between the data and the training job.

When storage mounts the bucket directly as a filesystem, the workflow changes in ways that actually matter. Training scripts point at a mount path instead of a download URL. Preprocessing writes back to the same path. Checkpoints flush without pausing the run. A new team member or a parallel agent mounts the same filesystem without copying anything. The bucket is the source of truth, and everyone works from it directly.

For inference: model weights load at first request from the NVMe cache. No pre-download step before the endpoint is ready. The cache warms on first access, and everything after that hits NVMe latency, not object storage latency.

For agent workflows: a persistent filesystem means an agent can read, write, and run commands against training data across sessions using the same workspace, without re-ingesting state on every run. That's a meaningfully different model of agent infrastructure compared to per-session scratch space.

So here's a quick gut check. If a storage change requires a data engineer to rewrite a pipeline, the architecture is still treating storage as a passive lake. If training starts the moment a bucket is mounted, the storage layer has caught up to the workload. Most teams are still in the first camp, which is why the data-movement problem keeps showing up on every ML platform roadmap.