S3 Multipart Upload Mechanics

The client sends a CreateMultipartUpload request. S3 responds with an UploadId. That's Phase 1. Done.

Except the UploadId is not just a receipt. It's the anchor for everything that follows — like a keystone holding an arch together: pull it out and the whole structure falls. Every part upload, every completion call, every abort request has to carry that UploadId. Lose it, and your parts become orphans. They exist in S3's storage. They're just not attached to anything. There's no path to completion.

A few things get locked in at initiation and cannot be changed mid-upload:

• Metadata, ACLs, storage class, and server-side encryption are all set here. Not at completion. You don't get a second chance.
• In-progress multipart uploads have no expiry. S3 holds those parts indefinitely until you explicitly complete or abort.

That second point is the one that bites teams quietly. I've seen buckets with months of abandoned part data nobody knew was there, accumulating charges the whole time. The parts exist. They're stored. They just never became objects. Lifecycle rules that auto-abort incomplete multipart uploads after a fixed number of days are the fix. If you're not running those rules, there's a reasonable chance you're paying for ghost parts right now.

Phase 2: Uploading Parts, Part Numbering, and ETag Tracking

Part numbers are integers between 1 and 10,000. They encode position in the final object, not arrival order. Part 7 can land before part 2. S3 doesn't care. Assembly happens by part number, not by the sequence things showed up.

The thing that catches people: uploading a new part with an existing part number silently overwrites the old one. This is actually the intended retry mechanism. Part 4 fails mid-flight? Re-issue the upload with part number 4. The failed attempt gets overwritten. Clean.

But if your part numbering logic has a bug and you reuse a number for a different chunk of data, S3 will not warn you. It just overwrites. The final object will be wrong. You won't know until you read it back and something downstream breaks. I've watched that exact failure take hours to diagnose because the upload itself completed successfully — a silent wrong answer dressed up as a success.

When a part lands, S3 returns an ETag. You have to store every (part number, ETag) pair. They're required for the CompleteMultipartUpload call in Phase 3. Drop them and you're calling ListParts to reconstruct what you had.

Part size constraints worth knowing cold:

• Minimum: 5 MB per part (the final part can be smaller)
• Maximum: 5 GB per part
• Practical floor for very large objects: total file size divided by 10,000. That's your minimum viable part size before you hit the part count ceiling.

One thing that doesn't get enough attention: in-progress multipart state persists across sessions. The UploadId and per-part ETags survive a client restart. You can pause an upload, bring the process back up, and pick up from where you left off without re-sending data that already landed.

Phase 3: Completing the Upload and How S3 Assembles the Object

The client sends a CompleteMultipartUpload request with the UploadId and the full ordered list of (part number, ETag) pairs. S3 concatenates the parts in ascending part-number order. That list is what determines the final byte sequence. Upload order is irrelevant. Part numbers are everything.

When assembly finishes, S3 returns an ETag for the new object. That ETag looks different from what a single-part PUT returns. It's computed as an MD5 of the concatenated binary part MD5 digests, hex-encoded, with a dash and the part count appended. So a six-part upload ends with something like -6 in the ETag. This matters later when you're comparing ETags across providers and things stop matching in ways that are inexplicable.

Two things to hold onto:

• Until CompleteMultipartUpload succeeds, the object does not exist in the bucket. Nothing is visible mid-upload. No partial object. Nothing accessible.
• AbortMultipartUpload frees storage for all parts uploaded so far and invalidates the UploadId. If an upload is dead, abort it explicitly. Don't wait for lifecycle rules to clean it up if you already know it's gone.

How Part Size Choices Shape Throughput and Overhead

Five MB parts are technically legal. For most workloads, they're also a bad idea. Every part is a separate HTTP request with its own overhead. At 10,000 parts of 5 MB each, you've got 10,000 round trips for a 50 GB file. That adds up fast.

The practical sweet spot most teams land on is 16 to 64 MB per part. It balances parallelism against per-request cost without pushing you toward the part count ceiling. On high-bandwidth connections, parts of 100 MB or more start making sense. Fewer round trips, less overhead.

But part size alone is not what makes multipart fast. Parallel uploads are where the real throughput comes from. Uploading multiple parts simultaneously is the actual lever. Part size tuning without concurrency tuning is leaving most of the gain on the table. The two work together — like a car engine and a transmission: one without the other and you're going nowhere efficiently.

One thing worth building around: objects uploaded with multipart can be fetched using byte-range GETs that align to the original part boundaries. When downloads use those same boundaries, you get parallel reads without reassembly overhead. The parallelism runs in both directions, and if you design for it upfront, you get faster reads essentially for free.

Prefix Partitioning and Request-Rate Limits That Cap Multipart Throughput

S3 routes requests to internal partitions based on object key prefix. Similar prefixes cluster on the same partition. If all your upload keys look like uploads/2025-07-01/file-001, uploads/2025-07-01/file-002, and so on, you're probably hammering a single partition with every concurrent upload. Per-upload parallelism can be perfectly tuned and it won't matter. The bottleneck just moves upstream.

The fix is randomness in your key prefixes. A hash prefix, a random hex string, anything that scatters keys across partitions. It feels weird to name objects something ugly and unsortable. Your throughput will not share that concern.

A few other levers worth knowing:

• Concurrency limits are configurable in most S3 clients, but setting them too high triggers throttling. There's a ceiling imposed by provider rate limits, and hitting it means retries and latency rather than speed.
• Transfer Acceleration routes uploads through AWS edge locations. Worth evaluating when clients are geographically far from the S3 region.
• VPC endpoints cut latency and data transfer costs for EC2-to-S3 traffic. If your workload lives inside AWS and you're not using a VPC endpoint, something is being left on the table.
• S3 Express One Zone is worth a look for latency-sensitive paths where partition-level isolation matters.

Failure Handling: Retrying Parts Without Restarting the Upload

This is the part of multipart upload that actually earns its complexity. A part that fails in transit doesn't break the upload. The UploadId stays valid. You re-issue the UploadPart call with the same part number. The failed attempt gets overwritten. You keep going.

What the client has to track is which parts have returned ETags. An ETag means S3 acknowledged the part. No ETag means the part hasn't succeeded. The CompleteMultipartUpload call requires ETags for every part in the final object, so gaps show up at completion, not during the uploads themselves. That gap between when a part fails and when you discover the problem is surprisingly wide if your tracking isn't tight.

If the process crashes and local state is lost, ListParts is the recovery tool. You query S3 for all parts associated with the UploadId, compare what came back against what the final object needs, and retransmit only the gaps. Parts that already landed stay put. The UploadId doesn't change. You resume rather than restart.

The failure mode lifecycle rules are built to catch is simpler and messier: the process crashes after initiation but before anything calls Abort or Complete. Parts sit there. They accumulate storage charges indefinitely. Setting lifecycle policies to abort incomplete multipart uploads after a fixed number of days is not optional defensive practice. It's just standard practice.

Multipart Upload in AI Training Pipelines: Checkpointing and Dataset Throughput

AI training workloads produce two distinct write patterns, and they stress multipart upload differently.

Checkpoint files are multi-gigabyte objects written frequently during training runs. A failed write at this scale is expensive to retry from scratch. Multipart's retry granularity changes the math. A failure mid-checkpoint means retransmitting the failed part, not starting the entire checkpoint over from byte zero. The difference between retrying one 16 MB part and retrying a 10 GB file is where the protocol's failure handling pays off most directly.

Dataset shards are the read-side counterpart. Shard sizes in the 100 MB to 1 GB range with sequential access patterns deliver better throughput for training I/O workloads. Writing those shards is a multipart upload problem. Reading them benefits from byte-range GETs aligned to part boundaries, which is the parallel GET pattern from Phase 3 showing up again in a different context.

The risk that makes all of this matter is GPU starvation. When storage can't sustain write throughput during checkpointing, or read throughput during data loading, expensive compute sits idle. That's the cost that actually shows up in budgets. The S3 API is stateless and horizontally scalable in ways that match how training fans out across many GPU processes issuing concurrent uploads.

On the self-hosted side, there's real published work from the Ceph community showing that objects above roughly 32 MiB with multipart enabled scale close to linearly across nodes. At a certain point, aggregate PUT and GET throughput hits the ceiling of the network interfaces, not the protocol. The protocol is not the bottleneck. The network is. That's actually a good place for multipart upload to stop being the problem.

S3 API Portability and Where Multipart Compatibility Gaps Appear Across Providers

The S3 API launched in 2006. Multipart upload is now implemented by effectively every object store worth naming: AWS, GCS, Azure Blob through compatibility layers, Cloudflare R2, Backblaze B2, MinIO, Ceph, and more. Code written against the S3 multipart API transfers to S3-compatible stores without changes. The API is the portability layer.

"Compatible" is doing a lot of work in that sentence, though.

Multipart upload is not optional compatibility. Iceberg, Spark, and other data lakehouse engines use it to write large Parquet files. An object store that implements 80% of the S3 API will fail in production even if demos pass. The failure shows up when a real workload tries to write a real large object, not during a demo writing a 10 MB test file. Put another way: a chain is only as strong as its weakest link, and a storage layer is only as compatible as its worst-supported operation.

ETag divergence is the most concrete cross-provider gap. AWS computes the multipart ETag as an MD5 of concatenated binary part MD5 digests, hex-encoded, with -N appended for the part count. Other providers compute it differently. If your integrity verification compares ETags across providers, you'll get false failures. The ETags are not comparable. This is documented. It's known. It still catches teams who treat ETags as universal checksums.

Cross-provider throughput variance is also real and significant. Multi-threaded upload throughput for the same file can vary by several hundred percent across providers depending on part size, concurrency, and region proximity. Part size and concurrency tuning help, but they don't close that gap. Provider-level differences are a separate variable entirely.

The practical response is a storage-agnostic architecture that mounts S3, GCS, R2, and Azure Blob through a common interface. Application code stays insulated from provider-level quirks. The multipart mechanics still operate at the protocol layer. You get the protocol's benefits without coupling your application to any single provider's implementation details, including whatever changes when that provider's maintenance status shifts.