Concurrent Write Patterns on S3

S3 picks a winner when two writers hit the same key at once, and it does not ask permission first. Nobody, not S3 and not your application, can predict which request actually arrives last. Network latency scrambles the order requests get received in, and it scrambles the order acknowledgments come back in too, so the outcome of a concurrent write race is non-deterministic from where your application sits.

This gets confused with a consistency problem, and it is not one. Whoever wins the race becomes instantly readable. The race itself is still a coin flip.

Add one more wrinkle: S3 only ever coordinates within a single key. There is no built-in way to update two keys as one atomic move. If your workflow needs a data file and a metadata entry to change together, S3 will not hold your hand through that. You own that coordination problem. None of this is a defect to file a ticket about. It is the tradeoff that makes S3 scale the way it does, and it is the starting line for every pattern that follows in this piece.

What S3 conditional writes guarantee

S3 shipped two primitives that change this picture substantially: write-once conditional writes in August 2024, and compare-and-swap conditional writes in November 2024.

The If-None-Match: * header tells S3 to accept a PUT only if nothing already exists at that key. Somebody else already wrote there, and S3 responds 412 Precondition Failed. That is a clean signal: your write lost, and you know it immediately.

The If-Match header works the opposite direction. Somebody changed it in between, and the ETag no longer matches, and your write fails with the same 412. Both headers cost nothing extra. Standard request pricing applies even to the ones that fail.

Bucket owners can force every writer to play by these rules, using the s3:if-none-match and s3:if-match condition keys in bucket policy. Skip the header, and the request gets denied with a 403. Without that enforcement, one careless writer who forgets the conditional header can slip past every guarantee the rest of the system is relying on.

What these headers do not do is reach across keys. If your application needs a data file and a registry entry to move together as a single unit, conditional writes give you the building blocks, but you still have to sequence them and plan for what happens if the first succeeds and the second does not.

The write-once pattern: preventing duplicate uploads with If-None-Match

If-None-Match: * earns its keep whenever the rule is simple: this object gets written exactly once, never twice. A pipeline that fails partway through an upload can just try again: if the object already landed, S3 will say so with a 412, preventing a duplicate from slipping through.

Picture a data lake fed by several ingestion pipelines, all writing partitioned Parquet files. If a previous attempt already succeeded, S3 returns 412, and the pipeline knows to move on rather than write a duplicate copy of the same partition. That is the whole trick: deterministic keys plus a guaranteed existence check equals safe retries, no coordination service required.

This pattern has a clear edge, though. You need If-Match instead, which is the subject of the next section.

One more thing built into the design from day one: a 412 is information the application can act on. It tells the application either "this already happened, carry on" or "something unexpected is sitting here, go look." Treating 412 as a coordination signal is what makes this pattern pleasant to operate.

ETag-based compare-and-swap: optimistic locking for shared mutable objects

Shared objects that need to change over time call for a different move: ETag-based compare-and-swap. This turns S3 itself into the referee for shared mutable state, and it removes the need for a DynamoDB lock table in most single-key update situations.

The sequence runs like this: read the object and note its current ETag, compute whatever the new version of the object should look like, then issue a PutObject with If-Match set to that captured ETag. If the write goes through, the update was atomic as far as the application needs to care. If it comes back 412, somebody else wrote to that object in the meantime, so the application re-reads the fresh state and tries again.

AWS has demonstrated exactly this pattern for a dataset_registry.json file sitting in a data lake metastore, where multiple pipelines update a shared registry without any external lock service, each one racing against the current ETag of the registry. An ETag is a reliable stand-in for version state, as long as the application treats it as opaque and re-reads it fresh before every attempt rather than reusing a cached value from three retries ago. Cache it, and the whole scheme falls apart quietly.

Compare this to what the same job used to require: that is an entire service removed from the architecture diagram.

Optimistic locking like this works best when conflicts are the exception, not the rule. At that point, the fix is not a bigger hammer, it is a different object layout, which the prefix sharding section covers later.

When per-key coordination is not enough: cross-key atomicity and the registry update problem

Plenty of real workloads need two different keys to update together, as one logical unit, and S3 has no built-in way to make that happen. A dataset file goes in one place. A metadata registry entry describing that dataset goes somewhere else. Both need to be correct and both need to be in sync, but S3 was never built to treat them as a single transaction.

The AWS multi-writer pattern handles this by sequencing, not magic. First, write the dataset file using If-None-Match, write-once, so duplicates are impossible. Only after that write succeeds does the application update the registry, using If-Match against the registry's current ETag. The order matters: write the data first, confirm the registry second. A crash between those two steps leaves a real file sitting on S3 with no corresponding registry entry, a state the application has to notice and clean up.

The registry carries real weight here, because it is the map every downstream reader consults to know what data actually exists. The two together, only if the application enforces the sequence.

It is a direct consequence of S3 being an object store rather than a transactional database, and that tradeoff is what lets it scale the way it does. Engineers who accept that constraint design for detectable partial failure. The next section covers how entire table formats were built specifically to absorb this problem so individual engineers do not have to solve it from scratch every time.

How Delta Lake and Apache Iceberg absorb the coordination problem above the storage layer

Delta Lake and Apache Iceberg both solve cross-key coordination, but they reach for different tools to do it.

Delta Lake leans on the name of its commit log file as the coordination mechanism. A writer trying to commit version 11 of a table attempts to create a file named 00000000000000000011.json inside the _delta_log directory. If another writer already created that exact file, the attempt fails outright, and the losing writer re-reads the latest state and tries the next version number. That scheme depends entirely on the storage system being able to do an atomic "create this file only if it does not already exist," which happens to be precisely what S3's If-None-Match: * now provides natively. Before conditional writes existed, running Delta on S3 meant standing up a separate coordinating service just to fake that atomic behavior. Delta Lake 4.0, released June 9, 2025, pushed this further with coordinated commits (shipped in preview), backed by a DynamoDB commit coordinator, enabling reliable writes across multiple clouds and multiple engines at once. Delta Lake 4.1.0, released February 26, 2026, added conflict-free feature activation, so turning on something like deletion vectors or column mapping no longer blocks every other writer mid-flight.

Iceberg takes a different road. No put-if-absent behavior at the object store level is required.

That structural difference has a real consequence for anyone choosing between platforms. Neither approach is the better one in the abstract. The right pick depends on the catalog already in use and where the workload actually runs.

Prefix sharding and object layout strategies for high-throughput concurrent writes

Not every concurrency problem needs a coordination header.

The classic contention trap looks like this: many writers all appending to one shared object, a single log file or one shared manifest. The fix is often structural rather than clever: give each writer its own distinct key to write to, and merge or compact those separate objects together later, asynchronously, well after the write pressure has passed.

Prefix sharding is the general version of this idea. Spread the key namespace out so concurrent writers land on different prefixes. Time-partitioned schemes, something like year=2024/month=01/day=01/, do this naturally for append-style workloads. Each writer gets its own file under its own prefix, a simple If-None-Match per file is all the protection needed, and no shared coordination object ever enters the picture.

That said, this move has a real cost that appears later, when the dataset is read back. For workloads where readers are constantly asking "what does the whole dataset look like right now," the merge cost at read time can eat back everything the simpler write path saved.

The AWS multi-writer pattern bakes this thinking directly into its object layout, keeping data files under /datasets/ separate from the shared registry under /metastore/. That separation is deliberate. It keeps the write-once hot path and the compare-and-swap hot path from ever tangling with each other, so each coordination mechanism only ever has to do its own job.

Application-level locking and S3 versioning

Conditional writes have not made every older technique obsolete, and some of them still earn their place rather than being leftovers from before S3 got smarter.

Application-level locking through DynamoDB still works the way it always has: a writer grabs a lock entry in a DynamoDB table keyed to the S3 object it wants to touch, does its write, then releases the lock. For a single key being updated by multiple writers, conditional writes now handle this job more simply. Any team keeping this pattern needs to manage lock TTLs carefully, since a crashed writer holding a lock forever is the kind of failure conditional writes sidestep.

S3 versioning solves a different problem. Nothing disappears. The cost is storage, and it only grows: every version retained on a frequently written object adds up, so lifecycle policies to expire old versions are not optional, they are the only thing keeping that cost in check.

S3 Object Lock is its own separate tool, built for a different job. It enforces WORM, write-once-read-many, semantics, blocking deletion or overwrite for a set period, a variable period, or indefinitely. That serves compliance and retention requirements. It has nothing to do with coordinating concurrent writers and should not be reached for as if it did.

Where S3's coordination model still falls short: latency, external dependencies, and the write-interoperability gap

Conditional writes and table-format coordination handle correctness for most concurrent write workloads, but three real constraints remain, and ignoring them means pretending S3 is now a universal concurrency solution.

Latency is the first one. S3 runs over HTTP, and even with strong read-after-write consistency, the round trip of a conditional write, read the object, compute the new state, write it back, costs noticeably more time than block storage or a local filesystem would. That cost is small on any single operation, but a long training run doing repeated checkpoint cycles pays it on every cycle, and it becomes a real drag on throughput. High-performance caching layers mounted as POSIX filesystems over S3-compatible buckets address this directly: reads get served from NVMe cache at sub-millisecond latency, writes flush back to the bucket asynchronously, and the object store stays the source of truth the whole time, without forcing teams to build separate pipelines just to stage data before a training run can even start.

External dependencies are the second constraint, and they are more subtle than they first look. Conditional writes were supposed to remove the need for an outside coordination system, and in the single-key case, they genuinely do. Layer a table format on top, though, and some version of that external dependency quietly comes back.

The third constraint is interoperability between writers themselves, and the practical answer is discipline: route all writes to a given table through one format, one protocol, one source of truth for commits.

A decision framework for matching the coordination pattern to the workload

Every pattern this piece has covered answers to the same three questions, asked about any given workload: what is actually contending for the same key, how often do real conflicts happen, and what does the write path need in terms of latency and durability. Answer those three honestly, and the right pattern mostly picks itself.

A single shared object, a registry, a manifest, a config file, updated by more than one writer but rarely fought over, calls for ETag-based compare-and-swap using If-Match, retrying cleanly whenever a 412 comes back. No external coordination service earns its keep here, and this is the exact shape of the metastore registry update problem described earlier in this piece.

Contention gets heavier when many writers hammer the same key frequently. Where a single shared object genuinely cannot be avoided, application-level locking with carefully managed TTLs becomes a reasonable fallback.

Datasets already managed by a table format should lean on that format's own concurrency protocol instead of layering custom coordination logic on top of it. Mixing writers from different table formats on the same underlying table defeats both systems' conflict detection at once, and should be avoided.

Latency-sensitive work, AI training checkpoints firing constantly, inference caches getting hit at high frequency, is a separate category entirely, because pure S3 conditional writes may simply not deliver the throughput or latency consistency that sustained checkpoint load demands. A POSIX-mounted caching layer sitting in front of the S3 bucket addresses that gap directly: it removes the separate download step, serves reads from NVMe cache, and flushes writes back asynchronously, all while the bucket remains the actual source of truth and without requiring a team to build its own data-movement infrastructure from scratch.

Compliance and auditability needs sit outside this coordination discussion and should be treated as overlays on top of it. Enable S3 versioning when the requirement is proof that no data silently vanished. Reach for S3 Object Lock when the requirement is immutability across a defined retention period. Neither one coordinates concurrent writers, and both can be layered on top of whichever pattern above actually fits the workload.