S3 Data Lake Architecture for AI Feature Pipelines

S3 was built to be cheap, durable, and decoupled from compute, so that analysts could run batch queries without worrying about where the bytes lived. What's changed is who's asking for the data and how fast they need it back. The AWS data lake whitepaper frames S3's whole value proposition around separating storage from compute and centralizing data in one multi-tenant architecture, and both of those design goals were built for batch analytical throughput, not for the low-latency, write-heavy, agent-driven traffic that AI feature pipelines now throw at it. A warehouse built to store pallets efficiently doesn't know, just by existing, how to run a same-day delivery operation out of that same building. Three forces are converging to force that change anyway: GPU-native data processing, lakehouse query engines fast enough to serve at millisecond latency, and object-level metadata that scales with S3 itself. None of this is a slow bend in the existing road. It calls for retiring assumptions that shaped the last decade of pipeline design, not tuning them. This piece now works through what that means, zone by zone and format by format, tracing how the lake shifts from a parking lot between pipeline stages into the active substrate the whole system runs on.

What the CPU-bound ETL assumption cost AI pipelines

The classic ML pipeline moves data through five stages: ingest, clean, featurize, store, serve. Every one of those steps ran on CPU, except training itself. That made sense back when models trained occasionally and the real bottleneck was how fast you could pull data off disk. Training runs now chew through terabytes of preprocessed data per hour, and the CPU transformation stage, not the GPU cluster waiting on it, has become the slow part of the whole operation.

A large GPU cluster waiting on a data loader that serializes preprocessing across CPU threads can drop below 40% utilization during data-intensive phases, Vector Labs' structural analysis found. Vector Labs calls this a structural mismatch rather than something a few config tweaks will fix. Picture this: you rent a fleet of race cars, then feed them fuel through a single garden hose. The cars aren't the problem.

NVIDIA's RAPIDS ecosystem, and its cuDF library in particular, deals with this by keeping transformation work on the same GPU memory the training process already uses. That removes the PCIe hop CPU preprocessing requires, where data gets shuttled back and forth between CPU and GPU memory on every batch.

This has a second-order effect on feature stores. Teams built feature stores in the CPU-bound era because computing features on the fly, during training, was too slow to put up with. The feature store existed as a workaround for a speed problem, not purely as a governance layer. GPU-native processing changes that calculation: complex augmentations, large-scale tokenization, and embedding generation can now run inside the training loop itself without dragging GPU utilization down. Vector Labs argues that for a growing slice of workloads, this makes the feature store layer less necessary, along with the schema management, freshness guarantees, backfill jobs, and offline/online consistency work that come with running one. That doesn't mean feature stores are going away; it means the question of whether a given workload needs one is getting renegotiated, not settled by default the way it used to be.

Open table formats as a governed feature layer on S3

S3 on its own is a very good filing cabinet. It doesn't know what's inside the folders, it can't guarantee two people editing the same file won't collide, and it has no concept of "this version of the data as of last Tuesday." Open table formats, Apache Iceberg, Delta Lake, and Apache Hudi, sit on top of S3 and give it that missing layer of intelligence: ACID-like transactions, schema evolution, time travel, hidden partitioning, and the ability for multiple engines to read the same table without anyone copying it first. S3 provides none of that natively.

DataForest frames this as the core move of the modern lakehouse: keep the raw data exactly as it arrived, but publish governed, structured versions of it for reporting, machine learning, and applications, all without forcing a rigid schema at the moment of ingestion. Multi-engine access matters especially for AI pipelines, since the same Iceberg table can be queried by Athena, Spark, DuckDB, and Arrow Flight SQL at once, with no duplicate copies and no separate serving store maintained for each tool.

Amazon S3 Tables pushes this a step further by making Iceberg a built-in part of S3 storage rather than a format bolted on top of it, through a dedicated "table bucket" type purpose-built for Iceberg rather than a general-purpose bucket pressed into service. AWS describes S3 Tables as the first cloud storage service with built-in Apache Iceberg support, with faster queries and a large jump in transaction throughput over standard S3.

That distinction isn't academic for high-frequency feature pipelines. Every streaming commit to an Iceberg table writes new metadata files, and if nobody manages the lifecycle of those files, the metadata piles up fast enough to become its own operational crisis. One documented case makes the scale of this painfully clear: an organization running more than 40 streaming feature groups watched its S3 bucket grow from a few gigabytes to over 50 terabytes of metadata in under a year, purely from ignoring Iceberg metadata lifecycle management. That's a bucket quietly turning into a hoarder's garage, one commit at a time, not a rounding error.

SageMaker Feature Store's Iceberg support gives teams a managed way to sidestep that trap. It supports streaming ingestion, scalable batch ingestion, and fine-grained access control through AWS Lake Formation, all directly on Iceberg tables. The practical wiring looks like this: Lake Formation governs the Iceberg catalog, Glue Data Catalog holds the metadata, and Athena or Redshift Spectrum queries it, with no data ever physically moved between systems.

Medallion zone design for AI pipelines versus analytics

The Bronze, Silver, Gold medallion structure is still the right skeleton for an AI feature lake. Bronze holds raw data exactly as it arrived. Silver cleans it, removes duplicates, and enforces types. Gold publishes the trusted, ready-to-use datasets that downstream consumers actually reach for. OneUptime's AWS implementation guide builds this as three physically separate S3 buckets, raw, clean, curated, each with versioning and KMS encryption turned on, and a consistent path structure (source, year, month, day) so Athena can partition the data efficiently.

That skeleton holds up fine for dashboards. But once AI training and serving become the main consumers, it starts to creak, and two zones in particular need a different set of decisions.

Bronze retention needs to think about reproducibility first. Training a model is an experiment, and like any experiment, someone eventually needs to rerun it and get the same answer. That means raw data has to stay immutable and versioned, with S3 Object Lock and bucket versioning treated as requirements rather than nice-to-haves.

Gold partitioning needs to flip its logic. Analytical Gold zones partition by date, because that's how BI queries slice the world. AI Gold zones should partition by the feature key instead (entity ID, model ID, experiment ID), so a point lookup for online serving can land on the right partition directly, instead of scanning through a pile of irrelevant dates to find it.

The small-file problem is a quieter trap buried in Silver that catches teams who haven't hit it yet. A computer vision training job might touch millions of individual image files, and at that scale, the filesystem spends more time just locating and opening files than it spends actually reading bytes. Metadata lookups become the bottleneck, not bandwidth. The fix is to pack small files into larger columnar containers at Silver, before training ever starts, rather than trying to patch the problem at training time when it's far more expensive to fix.

Gold also has to serve more audiences at once than any analytical lake ever had to: BI dashboards, ML feature pipelines, AI applications, and operational reporting, often pulling from the same tables simultaneously. That's why the multi-engine Iceberg access described earlier carries real weight at this layer, not just on paper.

Streaming ingestion changes the freshness contract that governs all of this. Kinesis Firehose can deliver straight into the raw zone, converting to Parquet on the way in, with buffering hints tuned to trade latency against file size. When Apache Kafka or Kinesis feeds directly into Iceberg tables at Silver, you no longer need a separate streaming layer at all, and the next section explains that same collapse.

The Lambda Architecture's Batch/Streaming Split in ML Serving

The Lambda architecture splits a pipeline into two parallel tracks: a batch path for historical features and a streaming path for real-time ones. Teams built this split because query latency from a lakehouse made it impractical to serve model features straight from a lakehouse at inference time, so they layered a separate low-latency serving store on top of the batch pipeline just to hit acceptable response times. That was a reasonable workaround given the tools available at the time. It appears in production AI systems as the main source of training/serving skew.

The mechanism is straightforward once it's spelled out: two code paths means two consistency models and two separate freshness SLAs to maintain. When the batch layer and the serving layer drift apart under any real write volume, the features a model sees at inference stop matching the distribution it was trained on. It's a structural property of running the same logic twice in two different systems and hoping they stay in sync forever, not a monitoring gap someone forgot to close.

Iceberg paired with fast query engines has closed the latency gap that justified the split in the first place, pushing point-query latency on lakehouse tables down into the tens of milliseconds. With Iceberg tables running on S3 Express One Zone, partitioned well on the feature key, and queried through DuckDB or Arrow Flight SQL with metadata caching turned on, single-entity lookups cost 5 to 20 milliseconds.

That range doesn't cover every workload. If a model's inference target is under 10 milliseconds, it still needs a dedicated key-value serving layer, because no lakehouse query engine gets there today. The real shift is that the set of workloads actually requiring a dedicated serving layer has narrowed considerably, not that it disappears everywhere, and a lot of pipelines running a Lambda split today are doing it out of habit rather than necessity. Collapsing to one lakehouse serving layer, with real-time ingestion feeding straight into Iceberg tables, removes training/serving divergence at the architecture level instead of leaving two teams to manage it by hand forever.

Object-level metadata, S3 Vectors, and feature discovery at lake scale

Feature discovery used to mean standing up a separate catalogue service just to answer questions like "which version of this dataset is current" or "where did this training shard come from." Object-level metadata querying, combined with S3 Metadata and AWS Glue Data Catalog integration, lets agents and pipelines run structured queries against an object's metadata directly, without ever opening the object itself. In practice, that means identifying the most recent model checkpoints for a given experiment, or filtering training shards by data source lineage, in under a second, with no separate catalogue sitting in the critical path.

That changes how agentic AI workflows actually operate. Instead of loading an entire inventory and lineage history into an agent's context up front, an agent can look things up as it goes, reading only the specific data it needs at each step. That's a meaningfully smaller and cheaper way to run discovery than front-loading everything and hoping the context window holds.

Amazon S3 Vectors extends the same logic to embeddings. AWS describes it as the first cloud object store with native support for storing and querying vectors directly, and positions it as cutting the overall cost of uploading, storing, and querying vectors by 90% compared to running a dedicated vector database. BMW's Cloud Data Hub platform, which already managed its curated datasets on S3 with Apache Iceberg, adopted S3 Vectors specifically because it slotted into that existing Iceberg architecture rather than requiring a parallel system to maintain.

Open table formats expose S3 as something SQL engines can query, but plenty of AI workloads, agents especially, would rather read and write features the way they'd read and write any other file on disk. A serverless filesystem layer like Archil addresses that by mounting an Iceberg table as a native filesystem, so agents can read, write, and checkpoint features directly without translating between file semantics and table semantics at every step. That's one option among several for closing the gap between how the lake stores data and how agents actually expect to talk to it, and it points at where this whole shift is heading: less a lake that gets queried on a schedule, more a live surface that pipelines, agents, and models all read from and write to at the same time.