Byte-Range Fetches for Parallel S3 Downloads
The core mechanic is blunt and simple. An HTTP GET to S3 can include a Range header that tells S3 exactly which bytes to return. Not the whole object. Just a slice. Something like bytes=0-8388607 for the first 8 MB.
S3 honors each of those requests independently, which means:
- Thread 1 asks for bytes 0 to 8 MB
- Thread 2 asks for bytes 8 MB to 16 MB
- Thread 3 asks for bytes 16 MB to 24 MB
All at the same time. Against the same object. Your aggregate throughput becomes the sum of all those concurrent connections, not the ceiling of any single one. Think of it like a highway: one lane moves one lane's worth of cars, but open up ten lanes and traffic flows. Parallel byte-range fetches are those extra lanes. That is a fundamentally different operating model than a plain GET.
There is also GET ?partNumber=N. If an object was uploaded via multipart upload, you can address each part directly by number without computing byte offsets yourself. Cleaner in theory. We will get to when it falls apart in practice.
One more thing worth holding onto before moving on: S3's per-prefix request rate baseline is 5,500 GET and HEAD requests per second. Spread your objects across 10 prefixes and that ceiling multiplies. The concurrency model scales horizontally across prefixes, not just within a single object. And if one chunk fails, you retry just that chunk. Not the whole object. That sounds like a footnote right now. It won't feel like one later.
Chunk Sizing: The Tradeoff Between Throughput, Cost, and Small-Object Coverage
AWS gives you a concrete starting point. For intra-region transfers from EC2 to S3, the guidance is 8 to 16 MB chunk sizes, with roughly one concurrent request per 85 to 90 MB/s of target throughput. To saturate a 10 Gb/s NIC, that math puts you at around 15 concurrent requests.
But chunk size is not just a throughput dial. It is also a cost dial, and teams almost always forget that second part.
Larger chunks mean fewer total requests for the same data. Fewer requests means lower cost, since cloud providers charge per request, not just per byte. Smaller chunks give you more parallelism headroom, which sounds great until you realize the tradeoff: more requests, more cost, more connection overhead to juggle.
Here is what teams most often miss entirely. If your chunk is bigger than the object itself, parallelism never engages. A 64 MB chunk size is completely useless for anything smaller than 64 MB. If you are working with a mixed workload (some giant model shards, some small metadata files) your chunk size needs to let even the smaller objects split into at least a few parts. Otherwise you have tuned for your biggest files and quietly abandoned everything else.
There is no universal right answer. The correct chunk size depends on your median object size, your target throughput, and how much you care about per-request costs. You have to know your workload before you can tune for it. Picking a number without doing that first is like choosing a shoe size without measuring your foot. You will get lucky sometimes. But eventually you will be limping around wondering what went wrong.
Part-Boundary Alignment and What Happens When Chunks Don't Match Upload Parts
When you upload an object via multipart upload, S3 stores internal metadata about where each part begins and ends. If your GET requests align to those same boundaries, you can use ?partNumber to address parts directly. Clean. Efficient.
When they do not align, S3 has to read across internal boundaries to fulfill your request. That adds latency. Not catastrophic. But real, and it compounds across thousands of requests in ways you will eventually notice.
The history of Transfer Manager is genuinely instructive here. Transfer Manager v1 retrieved parts the way they were uploaded. Objects uploaded as a single PUT had no internal part boundaries, so v1 could not sub-divide them for parallel download. No multipart upload meant no parallelism, full stop. Transfer Manager v2 dropped that constraint by switching to byte-range fetches, which work regardless of how the object was originally uploaded. It was a quiet change that made a large practical difference.
The key question is who controls the write path.
If you control it, align your upload part sizes to your intended read chunk sizes. Match the granularity of the PUT to the granularity of the GET. If you do not control the write path, use byte-range fetches instead of part-number fetches. Transfer Manager v2 and the CRT client handle this automatically if you let them.
The AWS SDK for.NET now exposes both modes explicitly. In a mixed environment where some objects came from one upload path and some from another, being able to select the right strategy per object is the kind of control that saves you from a lot of painful debugging later.
Concurrency Tuning: Scaling Connections to Actually Saturate the NIC
More connections mean more throughput, up to a point. That point is NIC saturation. Past it, you are adding overhead, not bandwidth.
The math is consistent: one connection per 85 to 90 MB/s of target throughput. A 10 Gb/s NIC needs roughly 15 connections. A 25 Gb/s or 100 Gb/s NIC needs proportionally more.
What does not scale for free is host resources. Every connection costs you file descriptors, buffer memory, and CPU cycles for TLS handshakes. Your practical ceiling is not just the NIC. It is the NIC plus whatever your host can actually sustain before things start falling over in interesting ways.
This is why connection pooling and DNS load balancing matter. The AWS CRT client implements both. Pooling cuts per-connection setup cost. Load balancing distributes requests across S3 endpoints, which starts to matter when setup overhead compounds at high concurrency.
The other piece worth paying real attention to is dynamic concurrency control. A fixed concurrency setting is a static guess, and static guesses age poorly. Meta's production experience shows exactly why. Checkpoint events created egress spikes, which caused congestion, which caused timeouts, which caused retries, which made the congestion worse. A feedback loop where client-side concurrency adjusts in real time based on application-level congestion signals was their way out. Not a number in a config file. A system that watches what is happening and responds to it. If you are running at meaningful scale, that is the model worth studying.
Retry Behavior at the Chunk Level and Why It Changes Failure Economics
Picture a 50 GB model checkpoint. You are 40 GB into a single whole-object GET when something drops the connection. You restart from byte zero. That is the failure model for a naive GET, and it is brutal in a way that only becomes obvious the first time it happens during a long training run at 2 AM.
With byte-range parallelism, only the failed chunk retries. The 39 other chunks that finished stay finished. You recover in seconds instead of minutes. That is not a subtle improvement.
Per-chunk retry also gives you per-chunk timeout thresholds. S3's first-byte latency is consistently low, typically somewhere in the 100 to 200 ms range. If a request is sitting well past that window, you do not have to wait for a full-object timeout to declare failure. You cut that chunk loose and reissue it. The CRT client handles this automatically, and AWS explicitly calls it out as a reliability improvement, not just a throughput one. That framing is correct.
For AI training pipelines iterating over large checkpoints or dataset shards, the distinction accumulates. A job that recovers in seconds instead of minutes, repeated across dozens of epochs, adds up to a lot of saved time that disappears invisibly if you never measure it.
One honest caveat: more concurrent requests means more in-flight state. If many chunks fail simultaneously, you can end up with a retry storm that amplifies request volume right when S3 is already stressed. Dynamic concurrency control is the answer. Throttle back during congestion rather than hammering S3 with retries and making things considerably worse.
SDK Tooling That Already Implements These Mechanics
You do not have to build any of this from scratch. Most of it already exists and is sitting there waiting.
AWS CRT-based S3 client handles automatic request parallelization, connection pooling, DNS load balancing, and per-part retry. On Trn1, P4d, and P5 instance types it is on by default. Everywhere else, opt-in. AWS tested it via the CLI on a trn1n.32xlarge against representative ML datasets and saw 2 to 6 times speedup with no code changes beyond switching clients. That is the return on doing almost nothing.
S3 Transfer Manager wraps multipart upload and byte-range fetches behind a simpler API and surfaces real-time transfer progress. v2 parallelizes downloads via byte-range regardless of how the object was originally uploaded, which removes the alignment headache described above.
AWS SDK for Swift S3 Transfer Manager extends the same multipart and byte-range approach to Swift-based inference and edge deployments, currently in developer preview.
Mountpoint for Amazon S3 is an open-source file client built on the CRT that exposes a bucket as a local POSIX filesystem. Standard file-oriented PyTorch data loaders get CRT-level parallelism without any S3-specific code. The cache is invisible to your training code, which is the goal.
Amazon S3 Connector for PyTorch benchmarked nearly 60% faster than FSSpecFileOpener for downloading an NLP dataset. That is not a rounding error. That is what native connector work buys you over generic file I/O.
How Access Patterns at the Dataset Level Determine Whether Any of This Tuning Matters
You can tune chunk size and concurrency perfectly and still have a slow pipeline. If your access pattern is wrong, none of the rest of it saves you. This is the part people do not want to hear, mostly because it means the problem is upstream of the thing they already spent time on.
The specific failure mode is small-file datasets. When every training sample lives in its own S3 object, your dataloader spends most of its time waiting for first bytes. Just waiting for S3 to respond to thousands of tiny requests. Byte-range parallelism within each tiny object cannot fix this. The bottleneck is per-request overhead, and you have a lot of requests.
The fix is format-level consolidation. Pack your samples into large tar shards. WebDataset is the common choice. A single GET then streams many samples sequentially, spreading per-request cost across all of them. The workload shifts from latency-bound to bandwidth-bound, and now your chunk and concurrency tuning actually has something to work with.
A Google Cloud benchmark found that switching to WebDataset format reduced epoch time from 432 seconds to 133 seconds compared to a conventional small-file layout. The format change, not the network tuning, was the primary driver of that difference.
The order of operations matters here:
- Fix the access pattern. Consolidate into large objects.
- Then tune chunk size and concurrency within those objects.
Doing step two without step one leaves most of the bottleneck untouched. Multi-epoch workloads make this worse, because every byte fetched in epoch one costs you again in epoch two. Which is exactly where caching comes in.
Where Caching Sits in the Pipeline and What It Does to Repeated-Epoch Workloads
Multi-epoch training is the norm. You iterate over the same dataset dozens or hundreds of times. Without caching, every epoch pays full S3 latency and request costs for data the instance has already seen. It is an avoidable tax, and it adds up faster than most people expect until they actually measure it.
The production numbers make the case plainly. Tigris tested a local S3-compatible caching sidecar against the same benchmark AWS ran in December 2025. On warm epochs, duration dropped by 5.7 times. The number of dataloader workers needed to saturate the GPU fell from 16 to 4. That second number is the one worth sitting with, because it tells you something about where the waste actually lives. Without caching, you are burning worker slots just managing data movement that a cache handles automatically.
Meta's production architecture took a tiered approach: read plan caches plus hard object caches on GPU hosts plus a regional distributed cache. The result was 80 to 90% cache hit rates, a 90% reduction in data ingestion latency, and no more manual cross-region dataset copying before training runs. CoreWeave's approach pre-stages frequently used datasets onto NVMe disks local to GPU nodes before the epoch loop starts. Subsequent epochs serve at NVMe latency, not S3 latency.
The filesystem interface matters here too. A cache that exposes a POSIX mount means your training code, checkpoint code, and preprocessing code all read from the same path. No S3-specific APIs. No special handling. The cache is invisible to the application. That invisibility is the feature.
Putting the Mechanics Together: What a Well-Tuned Parallel Download Pipeline Actually Looks Like
There are six decisions, and the order reflects how they depend on each other.
Dataset format first. Consolidate small files into large shards before touching anything else. WebDataset or equivalent. This is what moves you from latency-bound to bandwidth-bound. Nothing downstream matters until this is right. Skipping it is the most common reason teams tune everything else and still wonder why the numbers are disappointing.
Chunk size second. Target 8 to 16 MB for intra-region EC2 workloads. Adjust upward for very large objects where per-request cost dominates. Adjust downward if smaller objects also need to parallelize. Know your workload's median object size before picking a number. Do not guess.
Concurrency third. One connection per 85 to 90 MB/s of target throughput. Account for host resource limits alongside NIC capacity. Instrument it. Look at what is actually happening. Do not set it once and walk away assuming everything is fine.
Alignment fourth. If you control the write path, match upload part size to read chunk size. If you do not, use byte-range fetches. Transfer Manager v2 and the CRT handle this automatically.
Retry strategy fifth. Per-chunk timeouts and retries, not whole-object restarts. Use a client that already does this. At high concurrency, add dynamic concurrency control so retry storms do not compound your problems at the worst possible moment.
Caching sixth. Layer a local NVMe or regional cache in front of S3 for multi-epoch workloads. The warm-epoch difference is large and real. This is often the highest-leverage change available for iterative training, and it is consistently the last thing teams add when it probably should have been near the top of the list.
These decisions interact in ways that matter. Chunk sizing only has teeth after the format is right. Concurrency tuning only helps if alignment is correct. Caching multiplies gains the other decisions have already unlocked. Teams that treat this as one dial to turn hit diminishing returns fast and cannot diagnose why. Teams that work through the full stack consistently arrive at the same place: storage stops being the constraint, and compute becomes the bottleneck. That is where you want to be.
