S3 Data Lake Architecture for AI Feature Pipelines

More than a million data lakes now run on AWS, and almost none of them were built with AI training jobs in mind. That's the actual problem this piece is about. S3 became the default home for enterprise data because it's cheap, durable, and lets a dozen different tools query the same files without copying them. But feeding a GPU cluster is a different job than feeding a quarterly report, and the gap between those two jobs is where a lot of AI infrastructure money quietly goes to die.

The medallion zone model and what each layer actually does for AI workloads

Think of the medallion setup like a kitchen. Raw ingredients come in one door, get prepped at one station, then plated at another. Nobody serves a customer straight from the delivery truck.

The raw zone, or bronze, is that delivery truck. Every record lands exactly as it arrived, untouched, undeleted. No cleanup, no shortcuts. This is your audit trail, the thing you point to when someone asks "wait, what did the data actually look like six months ago?"

Silver is the prep station. Records get cleaned, joined, deduplicated, and schema gets enforced. This is the layer where data becomes something multiple teams can actually query without stepping on each other or dragging bronze's mess into their reports.

Gold is the plate that goes out to the table: training snapshots, feature tables, embedding indexes, shaped for whatever's about to consume them.

Here's why this matters more for AI than it did for a Tuesday morning dashboard refresh. Training reproducibility means being able to rebuild the exact dataset a model saw, down to the row. That only works if raw data never gets overwritten and gold snapshots are versioned like code. Inference wants low-latency reads. Training wants high-throughput scans. The same S3 objects can serve both, but only if your partitioning and caching are actually built around the zone they live in, not bolted on afterward.

Teams that skip silver tend to shove transformation logic straight into training scripts. Feels efficient for about a month. Then retraining breaks the moment someone tweaks a join, and nobody can figure out why, because the "transformation" was buried in a Python notebook instead of living somewhere shared and visible.

S3's real advantage here is that Spark, Flink, Athena, DuckDB, and a random Python script someone wrote at 2am can all read the same physical files. One copy, one governance boundary. And don't forget metadata and logs: lineage, data quality checks, feature usage tracking. Most teams treat this as homework they'll get to later. At scale, "later" is when governance quietly falls apart.

Open table formats as the mechanism that makes zones governable and reproducible

Raw S3 prefixes have no schema, no transaction boundaries, no history. Write directly to S3 without a table format on top, and you're one bad Spark job away from silent corruption or a training run nobody can ever reproduce again.

Apache Iceberg, born at Netflix for petabyte-scale tables, has become the cross-cloud standard by 2025. It's engine-agnostic: Spark, Flink, Trino, DuckDB, Snowflake, BigQuery, Presto, all reading and writing through the same REST Catalog API. That REST spec matters more than it sounds like it should. Any engine that speaks HTTP can create, read, and commit to Iceberg tables without needing a Java SDK, which means Python and Rust and Go clients all get a seat at the table (pun fully intended), and it works the same whether your data sits in S3, GCS, or Azure Blob.

Delta Lake is the natural pick if you're already deep in a Databricks-centric stack; the Spark integration and performance tuning are strong. Apache Hudi, which came out of Uber, is built for record-level upserts and deletes, so it's the right call for streaming ingest and CDC feature pipelines. Iceberg's edge shows up when your pipeline spans a bunch of different compute engines: hidden partitioning and schema evolution happen without rewriting the underlying data, which is the lowest-friction setup you'll find for a multi-engine AI stack.

For ML feature stores specifically, Iceberg's time-travel queries let you pull point-in-time correct features, which is the thing standing between you and label leakage. Snapshot tagging lets a team version a training dataset the same way they'd version code, tag it, branch it, come back to it in six months.

That's the single biggest win here: training data versioning through snapshots solves the reproducibility headache that's haunted ML teams forever. You can retrain against the exact historical slice a model originally saw.

There's a catch, though, and it bites people who don't see it coming. With high-frequency streaming writes, every commit spins up new metadata files. One production case saw a bucket grow from a few gigabytes of metadata to over 50 terabytes in under a year, from writes happening every few seconds, compounding quietly in the background. Lifecycle rules and compaction schedules aren't a nice-to-have at streaming scale. They're the difference between a healthy table and a metadata monster.

Where storage becomes the bottleneck that GPU spending cannot fix

Here's a number that should make anyone running a training cluster a little uneasy: in an MLSys 2022 analysis of over two million ML training jobs at Google, 62% had input pipelines that repeatedly stalled waiting on data. GPUs sitting idle more than half the time, not because compute was maxed out, but because storage couldn't keep up.

Idle GPUs aren't a shrug-it-off problem. When storage can't saturate a modern cluster, you're burning tens of thousands of dollars per node per year in capital and power on cycles that produce nothing. That's not a storage inefficiency footnote. That's a compute bill with someone else's name on it.

Object storage capacity and object storage performance get treated like the same problem, and they're not. S3's flat architecture scales to whatever size you throw at it. Fine. But the throughput ceiling for a single training job depends on request parallelism, network path, and caching, none of which happen automatically just because you picked S3. Over 70% of cloud-native enterprise data already lives in object storage, per MinIO's enterprise research. The bet on S3-backed lakes has already been placed industry-wide. The open question isn't whether to use it. It's whether the read path is fast enough for what you're asking it to do.

Global AI infrastructure spending topped $250 billion in 2025, with storage and networking growing almost as fast as compute itself. At that scale, storage inefficiency isn't a rounding error, it's a line item that gets bigger every quarter you ignore it.

Separating compute from storage was step one. It's not the finish line. The gap between the two has to be actively closed with caching, colocated storage tiers, and prefetching, which is exactly what the next section gets into.

Storage tier selection across the pipeline: hot, warm, and cold layers mapped to AI workload patterns

Large AI shops have mostly converged on the same three-tier pattern, and it maps pretty cleanly onto how often you touch the data.

Hot tier holds active training data and freshly computed features. This is NVMe-backed storage or S3 Express One Zone, sub-millisecond to low single-digit millisecond latency. This tier is what actually decides your GPU utilization number.

Warm tier is for feature store tables and silver-layer datasets that get queried regularly but not constantly. S3 Standard fits here, tens of milliseconds of latency is tolerable, and pricing runs somewhere in the $5 to $10 per terabyte per month range for object storage broadly.

Cold tier is for raw zone archives, old training snapshots, and anything sitting there for regulatory reasons. Glacier Deep Archive runs $0.99 per terabyte per month with a 12-hour retrieval window. Fine, as long as your pipeline genuinely never needs same-day access to it.

S3 Express One Zone deserves its own paragraph because the numbers are hard to ignore. Up to 10x faster access than S3 Standard, directory buckets supporting up to 2 million requests per second, and request costs cut by up to 80% compared to Standard, all at $0.11 per GB per month for storage. One AWS partner team reported cutting ML training epoch times by up to 35% just by moving to Express One Zone, no changes to model code, purely a storage layer swap. Run the math on an 8×p5.48xlarge cluster costing around $784 an hour: a 30% epoch-time cut saves roughly $23,520 a month in compute, against maybe $1,347 in extra storage cost. The storage upgrade pays for itself before you've finished reading the invoice. The catch is it's single-AZ, so it's the wrong call for anything that needs to survive an availability zone failure mid-training-run.

For feature pipelines specifically: online features, the ones served to inference in real time, belong in a fast key-value store, full stop, not S3. The lake is your offline store; the online store is a separate system fed from it. Offline feature tables live in warm or hot depending on how often models retrain. Historical snapshots kept only for audits or resurrecting an old model version belong in cold storage, with clear retrieval SLAs written down somewhere a human can find them.

None of this should live in tribal knowledge. Which prefix goes to which tier, when something gets promoted or demoted, who owns that call, all of it belongs in the same document as your zone definitions. Otherwise it's just whatever the last engineer decided felt right.

What real AI infrastructure looks like at the scale where S3 architecture decisions have measurable consequences

Uber runs 100 petabytes in an S3-compatible object store, backed by 50,000 CPU cores plus 5,000 ephemeral GPUs, a 10 petabyte distributed NVMe cache, and 10 terabytes per second of aggregate throughput. Compute scales from zero to 100% in five minutes. The payoff: a 65% cost reduction versus their old coupled architecture. Separating storage from compute and layering NVMe cache in between isn't two separate decisions, it's one design, and it's what makes the cost savings and the scale both possible at the same time.

Meta hit a wall at 50 GB/s because the CPU was the bottleneck, not the storage. After rolling out GPUDirect Storage across their research clusters, training data started streaming straight to GPUs at 192 GB/s, a 3.8x jump in training speed. The lesson there isn't about S3 throughput. It's about the path between storage and GPU memory, which is a completely different problem wearing the same disguise.

OpenAI has reportedly built a 50 petabyte WekaFS system with an S3 interface, sustaining 175 GB/s to 10,000 GPUs, with custom prefetching that dropped data loading from 30% of training time down to 5%. Prefetching is a software answer to what is fundamentally a hardware proximity problem, and that distinction matters if you're deciding where to spend engineering time.

Notice the pattern repeating across all three: NVMe cache or something like it sits between the object store and the GPUs. The lake stays S3-backed in every case. Nobody solved this by abandoning object storage; they solved it by closing the physical and logical distance between where the data sits and where the compute needs it.

If you're not running at Uber or Meta's scale, the shape of the solution doesn't change, just the size. Zone your lake, put a cache close to compute, and treat the S3-to-GPU gap as an engineering problem you actively manage, not a tax you just accept.

Closing the data-proximity gap without building custom data movement infrastructure

Most teams fall into the same default without meaning to: download training data to local disk before every job. That means every run starts with a data-movement tax, GPUs sit idle during the download, and now you're paying to store the data twice, once in S3 and once on local disk.

"Just read from S3 directly" sounds like an answer until you hit the access pattern mismatch. Raw S3 GET latency is fine for batch analytics running a handful of large scans. It falls apart when ML frameworks issue thousands of small sequential reads against big files. That mismatch is a big part of what produced the 62% stall rate in the Google study mentioned earlier.

Three approaches have actually stuck in practice. Prefetch pipelines: custom code that guesses what a training job will need next and stages it to NVMe ahead of time. Works, but it's a maintenance commitment, and it breaks the moment access patterns shift under it. Colocated high-performance tiers, like S3 Express One Zone or CoreWeave's AI Object Storage, announced in October 2025, which uses its Local Object Transport Accelerator technology for globally consistent access with no egress charges and claims over 75% lower storage costs for typical AI workloads. These work well inside a single provider's walled garden.

The third approach is a POSIX-compatible filesystem mount over object storage: present S3, GCS, R2, or Azure Blob as a real local filesystem to the training process. Reads hit an NVMe cache at sub-millisecond latency; on a cache miss, data pulls from the bucket and gets cached for next time. No download step, no duplicate copy sitting around, no code rewrite.

This matters especially for feature pipelines because training code written for local disk just works, unmodified, since a filesystem interface is what ML frameworks already expect out of the box. Agents and inference pipelines that need a stable, persistent workspace across sessions, rather than a fresh staging directory every run, benefit the same way. Archil is one option built specifically for this: it mounts S3, GCS, R2, or Azure Blob as a real POSIX filesystem, handles the hot path through NVMe cache, replicates writes before returning and flushes them asynchronously back to the bucket, keeps the bucket as the actual source of truth, and bills on what's actively cached rather than provisioned space sitting unused.

Whatever approach you pick, the bucket needs to stay the source of truth. Any setup that keeps a persistent copy of your data outside your own account trades a performance problem for a data residency problem, and that's a worse trade than it sounds like.

Worth a mention on the bulk side: S3 Batch Operations got a 10x performance boost announced at re:Invent 2025, now scaling to 20 billion objects per job, cutting jobs that used to take ten days down to about 24 hours. Great for large zone migrations or backfills. Not built for hot-path training reads, which is a different problem entirely.

Native vector and embedding storage as the emerging fourth zone in AI-first lake architectures

Bronze, silver, and gold covered the world of structured and semi-structured records. Vectors and embeddings don't fit neatly into any of them, and that's becoming its own architectural headache as RAG pipelines and semantic search move from side project to core infrastructure.

An embedding index isn't a table in the traditional sense. It's a high-dimensional object that needs approximate nearest-neighbor search, not a WHERE clause. Bolting a vector database onto the side of a lake works, but now you've got a fourth system to keep in sync with the other three, and every embedding refresh becomes its own mini data-movement project.

The direction things are heading: embedding storage and versioning start getting treated with the same discipline as gold-layer feature tables, snapshotted, tagged, tied back to the model version that generated them, so a RAG pipeline's retrieval results are exactly as reproducible as a model's training run. Teams that skip this step end up with the vector-store equivalent of the "transformation logic baked into training code" problem from Section 1: nobody can say for sure which embedding model produced which vectors, and debugging a bad retrieval result turns into archaeology.

There's no single winning pattern here yet the way Iceberg won for tables. What's consistent across teams doing this well is treating the embedding layer as a real zone, with its own lineage, its own versioning, and its own place in the governance conversation, rather than a side project living in whatever vector database had the best onboarding docs. Give it time. This is where table formats and feature stores were five years ago, right before everyone agreed on how it should actually work.