S3-Backed Databases and Table Formats

S3-backed databases and table formats solve one problem: object storage like S3 was built to hold data durably and cheaply, not to behave like a database. Apache Iceberg, Apache Hudi, and Delta Lake bolt database semantics, things like ACID transactions and schema changes, onto flat storage so engineers get the reliability of a database without giving up the cost and scale of S3.

Giving flat object storage database-like semantics

S3 was built for one job: keep data safe and available at massive scale, cheaply, for a long time. That job doesn't include knowing what a "row" is. It doesn't include letting two people update the same record at once without stepping on each other. S3 stores files. It doesn't know if those files make up a consistent, trustworthy table, and it was never asked to know that.

Say two different jobs write to the same dataset in a bucket at the same time, and that gap turns into a real problem. Nothing in S3 stops them from corrupting each other's work, because there's no referee checking who gets to write first or whether the two writes are even compatible. If a job dies halfway through writing new files, there's no undo button. The bad half-write just sits there, and the next query reads it as if it were finished and correct.

Query engines run into a quieter version of the same issue. Point a tool at a folder of raw Parquet or CSV files in a bucket: the engine has no way to know which files represent the current, correct state of the table. Old files, new files, half-finished files, they all look the same from the outside. Add a new column, and some files have it and some don't; the engine then has to either rewrite everything or live with the mismatch.

Open table formats exist to patch this hole. Iceberg, Hudi, and Delta Lake sit on top of S3 and impose the things a database normally gives for free: transactions that only become visible once they're fully committed, the ability to look back at a table as it existed yesterday, schema changes that don't require rewriting the whole dataset, and a system for managing partitions so queries don't accidentally scan ten times more data than they need to. None of this moves the data off S3. It just wraps metadata and rules around it.

S3 isn't just a warehouse for quarterly reports anymore; it's become the foundation plenty of teams train and fine-tune AI models on. When the underlying data has no consistency guarantees, that's a correctness problem, and in AI workloads, a correctness problem quietly becomes a "why is the model wrong" problem a few steps down the line.

Open table formats and their architectural differences

Three formats have led this space: Apache Iceberg, Apache Hudi, and Delta Lake, with Apache Paimon picking up steam as a fourth option. All three attack the same problem, giving flat storage database manners, but each one makes a different bet about how to do it, and those bets determine how each one performs under real workloads.

Apache Iceberg came out of Netflix, went open source in 2018, and graduated from the Apache Incubator in 2020. Its core idea treats the table format as a specification, a standard any engine can implement and commit against. Iceberg tracks the state of a table through a tree of snapshots and manifests, managed by a catalog, and any engine that speaks the catalog's API can commit changes with proper concurrency control. That means Spark, Trino, Flink, and others can all write to the same Iceberg table and trust that the catalog will keep things straight between them.

Delta Lake takes a different approach. It keeps a running log of every change, called the _delta_log, made up of JSON commit records and periodic Parquet checkpoints. Because that log describes the entire history of the table, the table's state can be rebuilt just by reading the log, no external catalog required. Delta Lake's multi-engine support is most mature inside Spark and the Databricks ecosystem, and for teams running a single engine, the log-based design is simpler to reason about: everything you need to know about the table lives in one place.

Apache Hudi rounds out the three major formats and takes the write path itself as the thing to optimize. Hudi was built around record-level upserts and incremental processing as core operations, not afterthoughts, which makes it a strong fit for streaming data and near-real-time pipelines, cases where individual records need to be updated or deleted constantly rather than in large periodic batches.

The three formats split along one clear line. Iceberg separates the specification from any single engine through a catalog API, so it's engine-agnostic by design. Delta Lake keeps the table's state embedded directly in its log files, which trades some flexibility for simplicity. Hudi builds its whole architecture around fast, frequent, record-level writes. None of these choices is wrong. They're different answers to the question of what kind of write pattern and what kind of engine flexibility a team needs most.

The catalog as the real boundary of the data platform

Picking a table format used to feel like the biggest decision in this stack. The catalog, not the file format underneath it, is where the real architectural weight sits now. The format increasingly becomes a detail the catalog can paper over.

Iceberg's REST Catalog Specification is a big reason why. It defines a plain HTTP API for managing tables: any tool that can make a web request, written in Python, Rust, Go, or JavaScript, can create, list, read, and commit to Iceberg tables without pulling in a Java SDK. That makes catalog access both language-agnostic and cloud-agnostic: the same API works whether the data sits on AWS, GCP, or Azure.

The convergence happening between Delta Lake and Iceberg makes the point even clearer. Delta Lake UniForm lets a single Delta Lake table expose Iceberg-compatible metadata at the same time, so tools like Snowflake, Databricks, Trino, and BigQuery can all read from that one table as though it were native to whichever format they expect. On top of that, Iceberg v3 (in Public Preview on Databricks Runtime 18.0+ as of April 2026, and generally available as of May 2026) picked up Delta Lake features directly: Deletion Vectors, Row Lineage, and a VARIANT type. The two formats are converging on the spec level. The format war that used to dominate this conversation is becoming less relevant than who controls the catalog sitting above it.

Amazon S3 Tables shows what that shift looks like in practice. It exposes the Iceberg REST Catalog API directly. S3 itself becomes a catalog endpoint. Any Iceberg-compatible engine, Spark, Trino, Flink, Athena, Redshift, Snowflake, can talk to it without any format-specific integration work, and AWS states this delivers up to 10x higher transactions per second compared to Iceberg tables stored in general purpose S3 buckets. Governance lives at this layer too: S3 Tables makes table-level access control, encryption, and lifecycle management first-class AWS resources, so teams no longer have to manage S3 bucket policies one table at a time.

Put together, these details point to a practical conclusion for teams building on this stack: the lock-in that actually matters long-term is about which catalog controls what engines can see, which policies apply, and which metadata gets trusted.

S3 Tables: AWS's fully managed Iceberg layer

Amazon S3 Tables takes everything described above and turns it into a managed product. It delivers fully managed Apache Iceberg tables, handling the operational grind, compaction, snapshot management, cleaning up unreferenced files, that teams running Iceberg themselves would otherwise have to build and babysit.

That operational layer runs continuously in the background. S3 Tables keeps optimizing Iceberg tables through compaction and maintenance on an ongoing basis, so query performance stays steady as data grows instead of slowly degrading the way unmaintained tables tend to. Teams with distributed users get automatic replication, which cuts down query latency for people working from different regions. Storage costs can drop too: S3 Tables supports an Intelligent-Tiering storage class that automatically moves infrequently accessed data to cheaper tiers, though it has to be turned on deliberately. By default, S3 Tables uses the Standard storage class.

On raw performance, AWS's own figures put S3 Tables at up to 10x higher transactions per second compared to Iceberg tables in general-purpose S3 buckets, which changes how teams plan capacity for write-heavy workloads.

None of this locks teams in on the compute side. Because S3 Tables expose the Iceberg REST Catalog API, they work with Spark, Trino, Flink, Athena, Redshift, Snowflake, and other third-party tools without modification. S3 Tables also supports Iceberg V3, which brings Deletion Vectors for faster data modifications and lower compaction costs. There's an AI-facing piece as well: MCP support lets AI agents and large language models interact with S3 Tables directly, so a natural-language question can turn into an analytics query without anyone writing SQL by hand.

The real shift here is about where engineering time goes. Writing compaction jobs and managing snapshot retention used to be a permanent line item on a data team's roadmap. A managed layer like this shifts that work onto AWS, giving teams back that time for the actual analytics and modeling work the data was collected for.

Where the table format layer breaks down

Table formats are a huge upgrade for analytical and batch work, but they have a ceiling. They don't fit workloads that need millisecond transactional response times, complex ad-hoc querying, or per-record access at the speed of an operational application. That's a property of the category, not a flaw specific to any one format.

The limits come from the architecture itself. Object storage has gotten fast, S3 Express One Zone offers consistent single-digit millisecond latency, but even that still adds round-trip overhead that purpose-built OLTP databases are designed to avoid. Iceberg and Delta Lake are built to make large sequential reads across many files efficient. They were never built to make a sub-millisecond lookup of one specific row fast, because that's a fundamentally different access pattern.

The maintenance work has a cost too. Compaction causes write amplification, the same background optimization that keeps a table performant also means data gets rewritten more than once, and that rewriting burns compute and I/O. S3 Tables automates that maintenance, but automating a cost doesn't erase it. It appears in workloads with high write frequency, where the rewriting happens more often.

Metadata can become its own bottleneck at scale. As the number of snapshots, manifests, and small files grows, catalog operations slow down. AI training workloads illustrate this well: a computer vision dataset made of millions of individual JPEG frames can end up with metadata lookups taking longer than reading the actual image data they're pointing to.

Table formats are the right tool for structured, schema-shaped data written in batches or streams. They're a poor fit for unstructured data, documents, model checkpoints, media files, which agents and training pipelines need to read and write with arbitrary, file-system-style access patterns. Databases and file systems are complements here, not rivals fighting over the same job, each covering ground the other wasn't built for.

What complements the table format layer for unstructured data and agent workloads

A single S3 bucket can back an Iceberg table and a POSIX file system at the same time. It's the actual shape the modern AI data stack needs to take, not an either-or decision between two competing designs.

There's a reason the file system matters so much here. Frontier models are trained disproportionately on bash and command-line tools, and agents built on those models overwhelmingly prefer reading, writing, and manipulating data through file-system commands over SQL or some custom API client. The file system is simply the interface these systems already know how to speak, which makes it the natural way to connect AI workloads to unstructured data sitting on S3.

That interface solves exactly the problems the table format layer can't. Unstructured data, documents, checkpoints, media, logs, has no schema to enforce in the first place, so a POSIX interface lets agents and training jobs work with it using the same commands they'd use anywhere else, no ETL step standing in the way. Agents working across multiple sessions need a stable workspace they can read from, write to, and run against repeatedly, something a table format, built around discrete batch commits, was never designed to offer. And because it's often unknown ahead of time whether an agent's task needs a few megabytes or a few hundred gigabytes, elastic, metered storage fits better than provisioning fixed blocks of capacity in advance.

One principle holds across both layers. The bucket stays the single source of truth. A storage layer sitting on top of S3 should never keep its own persistent copy of data outside a customer's account, and if access gets revoked, the data stays exactly where it was, on S3, still readable by whatever Iceberg-compatible engine the team was already using.

That's the complete shape of the stack. Structured, schema-bearing data runs through the table format layer, Iceberg, Delta Lake, Hudi, S3 Tables, for ACID guarantees and fast analytical queries. Unstructured data and agent workspaces run through a POSIX file-system layer mounted over that same bucket. Two interfaces, one store, no copies sitting around, and no pipelines moving data back and forth between them.