S3 Multipart Upload Throughput Tuning

S3 multipart upload exists because a single HTTP connection physically cannot move data as fast as a well-organized convoy of connections can. TCP throughput limits cap how much one connection can push, no matter how generous the pipe behind it is. Multipart upload is AWS's engineering answer: split the file, send the pieces over several connections at once, and stop waiting on one slow lane.

There's also a hard size wall that forces the issue. A single PUT request tops out at 5 GB per object, period. Multipart upload is the only route to objects up to 5 TiB, and plenty of ordinary workloads blow past that 5 GB line without trying: database backups, video archives, machine learning training sets, and the kind of sprawling log bundles that pile up in any production system.

The protocol itself draws three lines that no setting can move. Every part except the last one must be at least 5 MiB. One upload can hold at most 10,000 parts. A finished object can reach at most 5 TiB. These numbers come straight from the Amazon S3 User Guide, and they're the fixed walls of the room you're tuning in, not suggestions.

Tuned well, the payoff is real and measurable. Multipart uploads run from an EC2 instance in the same region as the destination bucket reach noticeably higher sustained throughput than a single PUT aimed at the same endpoint. Parallel parts use the available bandwidth instead of leaving most of it idle, and if one part fails, only that slice gets retried, not the whole object. That's the baseline the rest of this piece builds on: a mechanism built around physical limits, not a feature you switch on and forget.

AWS CLI parameters that determine upload throughput

Once multipart upload is the plan, four settings decide how well it actually performs: multipart_chunksize, max_concurrent_requests, max_queue_size, and max_bandwidth. Treat them as one shared budget, because that's what they are. Change one without checking the other three and the upload doesn't just underperform; it can jam or fail.

The AWS Knowledge Center lays out defaults and ranges for each. multipart_chunksize defaults to 8 MB, with 16-64 MB recommended once files cross 1 GB. For high-bandwidth connections that have the system resources to back it up, 20-50 is recommended, though max_concurrent_requests defaults to 10. max_queue_size defaults to a value in the thousands and should go up for batch jobs pushing many files at once. max_bandwidth defaults to unlimited, and the safer move for most setups is leaving it null and letting the CLI manage things.

The coupling between these four is where tuning goes wrong for a lot of people. If you push concurrency up without raising queue size, the worker pool starves, because the queue can't feed it fast enough. Pushing chunk size up without doing the arithmetic against the 10,000-part ceiling makes a big enough file simply fail to upload. Neither mistake shows up as "slightly worse": both show up as broken.

One more knob sits just outside this group: the multipart threshold, which decides the file size that triggers multipart behavior. It's a separate lever from the four above, and AWS recommends leaving it at the default for most use cases. Save the tuning effort for the parameters that actually move throughput.

Choosing part size: the constraint that governs everything else

Part size looks like a free dial, but it isn't. Two constraints squeeze it from opposite directions, and the safe range you're left with is whatever survives both.

From above, the part-count ceiling sets a floor on how big each chunk has to be. A 100 GB file split into 64 MB chunks is comfortably under the 10,000-part limit, plenty of margin. Split that same file into 8 MB chunks, and the math doesn't work: the file needs more parts than the protocol allows, the upload fails, and that's a hard stop. The AWS Knowledge Center's 16-64 MB recommendation for files over 1 GB exists specifically to keep practitioners away from that wall.

From below, network conditions set the practical choice you make within that safe range. If your connection is fast and stable, bigger chunks (toward 64 MB) cut down on per-part overhead and keep every concurrent worker busy with meaningful work. On a slower or shakier connection, smaller chunks (8-16 MB) spread the load better and, more importantly, limit what a retry costs. A failed 64 MB part means re-uploading 64 MB. A failed 8 MB part means re-uploading 8 MB. On a flaky connection, that difference adds up fast, even though smaller chunks cost you a bit more overhead when everything goes right.

There's a floor under all of it too: the 5 MiB per-part minimum. Going smaller than that buys nothing, and S3 will reject the completed upload if any non-final part falls short of it. One tuning guide from RONIN suggests starting an experiment at 100 MB chunks for large-file workloads and adjusting from there, which is a reasonable way to find your own sweet spot rather than assuming someone else's number applies to your file sizes and your network.

Part size isn't chosen, it's negotiated between a protocol ceiling you can't move and a network reality you can only partially control.

Setting concurrency: finding the saturation point rather than guessing at it

Concurrency has a point past which adding more workers stops helping. That point moves depending on the endpoint, the network path, and the instance type doing the uploading. Picking a number off a chart is a guess, not a tuning decision.

AWS's own recommendation, 20-50 concurrent requests for high-bandwidth connections, is a wide range on purpose. It's telling you the right number depends on your workload, not handing you a single answer. The way to actually find it is a sweep: run the identical workload at concurrency values of 4, 8, 16, 32, and 64, and watch where throughput stops climbing. That plateau indicates something in the path has saturated. It doesn't tell you what, only that more workers past that point are wasted effort.

And it's not free to overshoot. Cranking concurrency past the saturation point burns system resources without buying speed and can destabilize the transfer. At least one systematic benchmark found zero further improvement beyond 8 threads on a given path, which is a useful reminder that "more parallel workers" is not a universal law, it's a bet that pays off only up to a point specific to that setup.

Machine resources cap how far you can push this before the sweep even matters. The AWS Knowledge Center is explicit: check that the machine running the upload actually has the CPU, memory, and connection headroom to support the concurrency ceiling before raising it. A laptop asked to run 50 concurrent requests on a modest connection isn't doing science, it's just choking itself.

Fix chunk size first, then sweep concurrency against it. Smaller chunks paired with high concurrency pile up connection overhead. Larger chunks paired with low concurrency leave workers sitting idle while a big part crawls across the wire. The order matters: settle one variable before you start measuring the other.

How the CRT-based client changes the concurrency model

AWS now points practitioners toward the Common Runtime (CRT)-based client for large-scale transfers or complex workloads, and this isn't a cosmetic option sitting next to the standard one. It changes how concurrency gets managed underneath the CLI, including which numbers you type into a config file.

Switching to it takes one line: set preferred_transfer_client = crt in the CLI config, or run aws configure set default.s3.preferred_transfer_client crt. That's the entire setup cost, which is part of why knowing about it before a transfer actually needs it matters.

The CRT client swaps max_bandwidth for a different setting, target_bandwidth, and AWS's advice here mirrors its advice on the standard client: leave it null and let the CLI optimize on its own. Setting a target_bandwidth value by hand risks overwhelming either the network or the machine running the job, so the safer path is to trust the automatic handling unless there's a specific reason not to.

Because the switch is a config flag rather than a visible process change, confirm it actually took effect. The AWS Knowledge Center provides a debug grep specifically to check that the CRT client is actually running. Skipping that check means potentially tuning a client that was never active.

The practical line: large files and high-throughput, repeated workloads are where the CRT client earns its place. Small files and occasional, low-frequency transfers don't generate enough volume to justify the switch, so the standard client handles those fine. Treat the client choice as its own tuning decision: make it before you touch chunk size and concurrency, not as an afterthought bolted on once those numbers are already set.

Network path decisions that determine the practical ceiling before any S3 parameter is touched

No amount of chunk-size or concurrency tuning can fix a bad network path. Every part upload is an API call, so you pay the round-trip cost on a bad path thousands of times over during one large transfer. That accumulated cost becomes the dominant factor in total upload time, so it swamps whatever gains you get from a perfectly chosen chunk size.

Region proximity is the first thing to check, before touching a single S3 parameter. Testing latency to candidate regions up front catches a problem that no amount of parameter tuning can undo later: a meaningful round-trip difference, multiplied across thousands of part-level requests, turns into real wall-clock time lost.

For an EC2 instance already sitting in the same region as the target bucket, a VPC endpoint or the standard regional endpoint already rides the AWS backbone. Adding Transfer Acceleration on top of that adds an extra hop for no reason and can end up slower. Transfer Acceleration earns its keep on a different problem entirely: geographically distant transfers, where data enters the nearest CloudFront edge location and rides AWS's internal backbone the rest of the way to S3. Before you pay the added per-GB charge, AWS recommends you check the actual gain with the Transfer Acceleration Speed Comparison tool. You can turn it on with a single CLI command, aws s3api put-bucket-accelerate-configuration, paired with the accelerated endpoint format, but the cost comes on top of standard transfer pricing.

Below the region and endpoint decisions sits one more layer: OS-level TCP buffer sizing for EC2-based transfers. Tuning net.core.rmem_max, net.core.wmem_max, and the tcp_rmem and tcp_wmem vectors lets sockets absorb bursts of data that would otherwise stall the whole upload pipeline while the buffer catches up.

The plainest limit of all is physical network capacity, and it constrains everything else described here. A constrained office connection has a throughput ceiling no S3 setting touches. Diagnosing a slow upload starts with checking that ceiling, not with opening the CLI config.

Prefix partitioning as a request-rate prerequisite for high-throughput uploads

S3 enforces request-rate limits per prefix, not per bucket. If a perfectly tuned multipart upload routes every part through one prefix, it runs straight into that ceiling, and no SDK parameter, no chunk size, no concurrency setting reaches past it.

The fix is organizational rather than technical: spread uploads across logical prefixes, such as uploads/batch-a/ and uploads/batch-b/, instead of funneling everything into a single dated path. It's a filing decision as much as a technical one, closer to running multiple checkout lines at a store than making everyone wait behind one register.

This isn't advice everyone needs. S3 has handled sequential key names automatically for most workloads since 2018, so randomizing prefixes for the sake of it is no longer necessary for moderate throughput. The case for spreading across multiple prefixes applies only when a workload genuinely pushes past what a single prefix's request rate can support, so it rules out a lot of everyday upload jobs.

Multipart uploads make this relevant faster than it might seem, though, because each uploaded part is its own PUT request. A high-concurrency upload with thousands of parts can chew through a single prefix's request budget a lot quicker than the file count alone would suggest. The request volume hiding inside "one big upload" is bigger than it looks from the outside.

Benchmarking your actual configuration rather than trusting reference numbers

Every number in the sections above, the 16-64 MB chunk range, the 20-50 concurrency window, the plateau at 8 threads on one particular benchmark, is a starting point from someone else's setup, not a guarantee about yours. Chunk size, concurrency, client choice, network path, and prefix structure all interact, and they interact differently depending on the specific endpoint, the specific file sizes, and the specific network sitting between the two.

Testing a baseline is simple: create a file of known size, upload it, and time it. A 100 MB file should take you roughly a second on a 1 Gbps connection. If it takes 30 seconds instead, something upstream of the S3 configuration is broken, and no amount of parameter tuning fixes a problem that's actually a network or region issue. Measuring upload speed directly (bytes divided by elapsed time) turns tuning from guesswork into something closer to an actual experiment, repeatable across each change made to chunk size, concurrency, or client.

The reference numbers in this piece are a sensible place to start sweeping, not a destination to land on and stop. The only number that tells the truth about a given upload pipeline is the one measured against that pipeline's own file shapes, its own region pairing, and its own network path.