File Systems and Databases as Complements

Picture a planning meeting. Someone draws two boxes on a whiteboard: object storage with a table format bolted on, or a relational database. The team argues for twenty minutes about which one to "go with," as if choosing a database were like choosing a phone carrier. That argument is built on a bad premise. File systems and databases are two different tools built for two different kinds of work, not two brands competing for the same job, and asking which one wins is a bit like asking whether a hammer beats a tape measure. Both are mature, both have been around for decades, and both technically "store data," which is why people assume they're interchangeable. They aren't, because the real question a team should be asking isn't which primitive to pick, but which layer of the job each one is suited for, since in most real systems, both end up doing work at the same time, just not the same work.

What file systems and databases are each built to do

Start with file systems. They give you a durable, hierarchical place to put bytes, any bytes, without asking what those bytes mean first. Images, audio, model weights, log files: none of it needs a schema to land on disk. File systems also let multiple machines read and write the same files at once, which matters enormously when dozens of GPU processes all need to chew through the same training set simultaneously. That's a structurally different job than what a relational engine does. And because file systems speak full POSIX (atomic rename, file locking, mmap, fsync, hard links, symlinks), old code and tooling just work. No special software kit required. No client library to install. You point existing programs at the files and they run.

Databases solve a different problem. They impose structure (schema), they guarantee that transactions complete fully or not at all (ACID), and they let you ask precise, complex questions of structured data in ways a plain file store never could. They also handle the unglamorous but critical stuff: who can access what, how data changes over time, how queries get planned efficiently, how metadata gets tracked. A relational database is terrific at anything with a stable, well-defined shape, especially when you need every record to stay consistent with every other record.

The cleanest way to see the split: object storage deals in whole objects, fetched or written through an API call. A database deals in individual records, which it filters and joins through a query engine. Neither one is a bigger version of the other. They're different shapes of tool for different shapes of problem.

Where each primitive hits its structural ceiling

Every tool has an edge where it stops working well, and that edge tells you more about the tool than its marketing copy ever will.

File systems hit their ceiling on metadata. Take a computer vision training job that has millions of individual JPEG files. The storage system doesn't struggle to read the files once it finds them, but finding them is the bottleneck. Opening millions of small files one at a time turns into the actual bottleneck, not the bandwidth used to stream the bytes once they're open. A storage system can post great throughput numbers on a spec sheet and still choke in production, because the benchmark measured reading speed, not file-lookup speed, and those are two different problems.

Object storage runs into a different wall: latency and query expressiveness. It was built for bulk durability (store a lot, lose none of it, keep costs low), not for the millisecond response times or complex filtering that interactive applications demand. Its key-value design makes rich SQL-style queries awkward to express, the same way trying to write a novel using only emoji is technically possible but not what the format was made for.

Databases have their own blind spot: raw, unstructured, high-volume data. Cramming images, audio, video, or embeddings into a relational schema adds friction, and it usually leaves you maintaining a second copy of the data somewhere else, because the data never had a tabular shape to begin with. Databases are still excellent if the data fits neatly into rows and columns. They're just the wrong tool for storing raw bytes at scale, and no amount of schema cleverness changes that.

AI training is where the gap between primitives becomes a pipeline problem

AI training is where these structural limits stop being theoretical and start costing real money and real time. Training workloads combine three demands at once: massive scale, dozens of processes hitting the same files in parallel, and sensitivity to latency measured in milliseconds. Traditional block or object storage, built for bulk transfers and transactional writes, wasn't designed for shared, simultaneous access from a fleet of GPUs that are each burning through terabytes of data an hour and waiting on every file lookup to finish before the next training step can start.

Meta's experience is the clearest documented version of this collision. Over the years its storage system for large binary objects grew organically, with new layers added on top of old ones, service by service, and many of those layers kept their own separate metadata stores. That kind of layered, stateful growth is normal for infrastructure that started before anyone anticipated today's training loads. But metadata-access speeds that were perfectly fine for older spinning-disk use cases turned into showstoppers once the workload demanded millisecond access to flash storage. A lookup that added up to hundreds of milliseconds across regions didn't just slow one request down. So a single slow metadata response could stall the entire training pipeline behind it.

The failure reveals that when data is scattered across tiers (network storage here, object storage there, local disks, cloud buckets), every hop between tiers adds latency, and that latency compounds. A routine backup job kicked off during an active training run can noticeably drag down GPU utilization, because the storage architecture wasn't built to keep training and backup traffic from stepping on each other.

How the complementary model works in practice

The fix that works assigns each layer of the problem to the primitive built for it, and uses a format or catalog to tie the two together coherently.

Open table formats are the bridge. Apache Iceberg, Delta Lake, and Apache Hudi all add database-style behavior (ACID transactions, enforced schema, fast structured queries) directly on top of object storage. The object storage layer still does what it's good at: durable, cheap, massive-scale storage of raw files. The table format adds what object storage can't do on its own: transactional guarantees and structured query performance. Neither layer is trying to replace the other. So several different query engines can read the exact same underlying files at the same time, and each tool doesn't need its own private copy of the data.

This is the logic behind the lakehouse architecture: cloud object storage on the bottom for durability and cost, an open table format above it for transactions and schema changes over time, a catalog layer for governance and so people can actually find what's in there, and a flexible top layer that serves SQL users, analytics teams, and AI agents through one consistent interface. Apache Gravitino is one example of how the catalog layer gets built: it's a high-performance, geo-distributed, federated metadata system that keeps track of metadata across file stores, relational databases, and streaming systems all at once, and presents a single unified view to engines like Spark, Trino, and Flink. That kind of setup matters most if you run across multiple clouds with several catalogs already in place.

An earlier-stage pattern is also emerging: building AI operations directly into the database engine itself, so the query planner can reason about SQL operations and AI operations together instead of running them in two separate systems that never talk to each other. It's an emerging approach rather than an established standard, but it points at where the boundary between "database" and "AI pipeline" is headed.

What ties all of this together: a proprietary storage format tends to produce a second, shadow copy of your data the moment you need to do anything beyond the one thing that format was built for. If an open format sits in storage you control, you get one copy of the data, readable by search tools, analytics tools, and whatever engine your team already uses. A proprietary format forces a shadow copy, while an open format keeps everyone working from one source of truth.

Three organizations that built on this division of labor

ByteDance, through Volcano Engine, rebuilt its AI data stack around the Lance format, and that cut a model training pipeline from seven days down to one. That's a fivefold-plus drop in turnaround time, revealing that the bottleneck wasn't raw compute or raw storage. It was the coordination tax paid every time data moved between the file layer and the metadata and query layer. Unifying the format closed that gap directly.

turbopuffer took a different path: it uses object storage as the permanent, authoritative home for its search indexes, while query nodes lean on memory and NVMe SSDs as fast caches sitting in front of that storage. So storage capacity and search compute can scale independently, and they don't stay locked together. Its approximate vector index runs on SPFresh, a centroid-based design the team chose partly because it cuts down on round trips to object storage and reduces write amplification compared to graph-based alternatives. There's a real trade-off baked in here too: a cold query might need to pull index data before it ever reaches the fast cache, so applications with strict latency requirements need to design around that rather than assuming every query behaves like a warm one. What this reveals: treating object storage as the durable base layer and compute as the query layer is a deliberate choice with real latency trade-offs attached, not a default setting you get for free.

Twilio built its control plane around the same split. It stages critical data in object storage, then accelerates it locally for fast queries, so P99 query times drop under 5 milliseconds, and it rolls out automated multi-tenancy controls that propagate updates in minutes. What this reveals is the cleanest version of the whole argument: the file system layer holds the one authoritative copy of the data, the database layer provides the fast, structured way to query it, and each one handles its own piece of the job.

The Boundary at Agent Workloads and Unstructured Data

None of this stays clean at the edges. The division of labor gets messy wherever data is neither fully structured nor purely raw bytes, and three zones sit right at that fault line: multimodal training data, AI agent memory, and retrieval that has to happen in real time.

Start with the strongest pushback against treating object storage as a catch-all hot tier: active training runs generally need lower, steadier latency than S3-compatible storage can promise. Over a long training run that scans the same dataset repeatedly, that latency gap doesn't stay hidden. It gets more visible with every pass.

Multimodal data adds another wrinkle. Images, audio, video, embeddings, and the metadata describing all of it need to sit close together without either forcing everything into a relational schema that doesn't fit, or scattering the pieces across separate systems that someone then has to keep synchronized by hand.

Agent memory might be the hardest test of all. An agent needs context that survives across sessions and across parallel runs. A single-session scratch pad doesn't cut it, because the whole point of memory is that it outlives any one session. That context is a genuine mix: raw artifacts like files the agent produced or code it wrote, sitting alongside structured state like which tasks got finished, what the agent learned, how confident it is in a given answer. That's a workload that needs both primitives running at once, not one standing in for the other.

The file system handles the raw artifacts, the database layer handles the structured state describing them, and the two stay connected as separate, linked layers. That's also why AI is opening up work in fields like healthcare, finance, and government, where the underlying data was unstructured from day one and nobody had a clean way to use it until this split became standard practice.

Assigning work to the right primitive in a real architecture

With competition no longer the frame, the practical question for any team is which primitive actually fits a given layer of data, and where the line between them should sit. The file system layer is the right call whenever the data is unstructured or multimodal, meaning raw bytes, model weights, images, audio, or code, and forcing a schema onto it would only slow things down without adding anything useful. The database layer, or a table format doing the same job on top of object storage, is the right call whenever records need to stay consistent with each other, whenever the data already has a stable shape, or whenever someone needs to ask precise, structured questions of it quickly. So the boundary between the two should sit wherever raw storage meets structured meaning, and a table format or catalog can act as the connective layer so both sides stay in sync instead of drifting into separate, unsynced copies. And at the harder edges, agent memory, multimodal datasets, real-time retrieval, the right move is letting the file system hold the raw material and the database hold the structure describing it, connected deliberately rather than smashed together and hoped into alignment.