NVMe Caching Layers and S3 Read Latency

S3 is slow to read from because it was built to optimize for durability, infinite scale, and rock-bottom storage cost, not speed. This piece walks through why that tradeoff exists, where it costs you real money, and what an NVMe caching layer does to fix it without asking you to abandon object storage at all.

Why S3's read latency is a structural property, not a configuration problem

Nobody at AWS sat down and decided object storage should feel sluggish. S3 was built to survive almost anything: hardware failures, entire data center outages, decades of neglect. Every one of those guarantees costs something, and the bill comes due in the form of latency.

Every read request has to travel across the network, resolve through a metadata layer, and in many cases coordinate across multiple availability zones before it hands back a single byte. The cost of that coordination across the network and metadata layer is the price of the durability promise. The same machinery that makes S3 nearly indestructible is the machinery that keeps its time-to-first-byte in the tens to hundreds of milliseconds, depending on conditions.

S3 also plays a trick on anyone used to thinking in folders. There's no real directory structure under the hood. Prefixes just look like folders, the way a well-organized junk drawer "looks" like it has sections. Listing a big prefix namespace means the system has to do real metadata work, and that work adds up fast for any access pattern that browses or traverses rather than grabbing one known key directly.

The latency isn't just high, it's inconsistent, which calls for a different fix. A request might come back in 30 milliseconds one moment and take much longer the next, depending on backend load and conditions nobody on the client side can see or control. For a batch job running overnight, that kind of variance barely registers. For a synchronous workload that needs data delivered on a predictable clock, that inconsistency is the whole headache.

None of this makes S3 badly designed. It means S3 was designed for a different job than "feed a GPU in real time," and expecting it to do that job anyway is like being annoyed your filing cabinet can't also toast bread.

Where that latency becomes expensive: GPU utilization as the cost unit

The engineering detail turns into a line item on somebody's budget. The real cost of S3's latency in AI infrastructure is GPU starvation: compute hardware that cost a fortune, sitting there waiting for data it can't get fast enough.

Profiling work at Uber, Shopee, and AliPay found that data loading, the steady-state read throughput moving data from S3 into GPU memory during a training step, eats up roughly four-fifths of total AI training wall-clock time. Read that again slowly. The GPU, the most expensive thing in the building, spends most of its life waiting rather than computing.

Teams usually find this out the hard way, after the cluster is already provisioned and the bill is already running. Training jobs run at a fraction of the GPU utilization the hardware is capable of, because the storage underneath, whether that's NFS, an underpowered S3 setup, or old-fashioned spinning disks, simply can't keep pace. It's the equivalent of buying a sports car and then discovering the only road to your house is gravel.

Meta's experience with GPUDirect Storage across its research clusters is the clearest documented example of this. Before the architecture changed, PyTorch training jobs were losing more than a third of their total compute time just waiting for data to arrive. A third of a very expensive cluster was sitting idle.

Inference has its own version of the same problem, just on a tighter clock. Interactive applications need to respond within a budget measured in a few hundred milliseconds, total. If fetching vector embeddings or a KV cache from object storage burns through most of that window before the model has even started generating a response, the GPU's speed stops mattering. The application just doesn't work, no matter how fast the chip underneath it is.

What an NVMe caching layer does between S3 and compute

The fix is to put something faster in front of S3 for the data that gets read over and over, not to replace it. An NVMe caching layer sits between S3 and compute, absorbing the hot, frequently requested reads at local storage speed while S3 stays exactly where it was: the durable, cheap, permanent home for everything.

The mechanism is simple enough to explain at a dinner party. When the compute node asks for data and that data is already sitting in the local NVMe cache, it gets served in under a millisecond. That's a cache hit. When it isn't there yet, that's a miss, and the system fetches the data from S3 once, then keeps a warm copy on the NVMe drive so the next request for that same data skips the network entirely. Think of it as a barista who remembers your usual order, the first visit takes a conversation, every visit after that is instant.

The real upgrade comes from prefetching. A reactive cache only helps after something's already been requested once. A predictive one is a different animal: it watches the pattern of what the data loader is asking for, figures out what it'll want next, and loads that into NVMe while the GPU is still busy chewing through the current batch. Done well, that doesn't just shrink I/O stalls. It erases them.

One well-documented example of this in practice: a system that learns data-loader access patterns and prefetches small 4 KB blocks into local NVMe ahead of need. In an AWS validation of that approach, a remote compute cluster matched the throughput of a cluster co-located with its data within the first hundred or so training batches. After that short warmup, the cost of being remote basically disappeared.

The shape of a well-built cache tier is almost absurdly lopsided. In one documented architecture reference, the NVMe layer held only one to two percent of total data volume, yet it was absorbing the overwhelming majority of actual read requests, because hot data clusters together. Most of a dataset just sits there quietly. A small, well-chosen slice gets hit constantly, and that is the one worth keeping close.

Deduplication and compression at the cache layer add another benefit: fewer repeated trips back to S3 for the same hot bytes means fewer repeated egress charges, which changes the math on frequently-read data.

None of this is free of risk, and it's worth being straight about the failure mode. NVMe storage on a spot instance disappears the instant that instance gets preempted, cache and all. That's a reason to pair the pattern with asynchronous checkpointing, using something like PyTorch's torch.distributed.checkpoint, so losing a cache never means losing a training run. Keep your receipts, basically.

The cost trade-off that determines when an NVMe cache makes sense

An NVMe cache doesn't lower your total storage bill. It moves cost around: away from per-read latency penalties and egress charges, and onto a smaller, pricier local tier. Whether that trade is worth making depends on how often the data gets read.

S3 is cheap to store data in, month over month. NVMe, by comparison, costs a lot more per gigabyte once you factor in the hardware and the node it's attached to. On paper that sounds like a bad trade. But NVMe earns its keep by killing repeated egress charges on data you read constantly, and that's exactly where the economics flip for high-frequency workloads, like a training job that loops over the same dataset dozens of times.

Picture it like buying a one-ride subway pass versus a monthly pass. If you're riding once, the single ticket is obviously cheaper. If you're riding every day, the monthly pass wins by a mile. NVMe caching only makes sense for commuters, not tourists.

Elastic, metered caching sharpens this further. If the cache bills you based on what's actually hot rather than some fixed amount of provisioned capacity, cold data that only ever gets read once never ends up parked on expensive NVMe at all. The cache shrinks to match the working set, not the whole dataset, so teams aren't paying premium storage prices to babysit data nobody's touching.

This isn't a niche concern. The demand behind it is large enough to be reshaping a market: Introl's analysis projects the AI-powered storage market growing from tens of billions of dollars to hundreds of billions over the coming decade, and that growth is being driven by precisely this bottleneck.

How the pattern is implemented across software-defined and hardware-accelerated approaches

The NVMe caching pattern appears at multiple layers of the stack, and these layers complement each other rather than compete. They're aimed at different parts of the same bottleneck, the way a good kitchen has both a sharp knife and a food processor.

At the software layer, distributed caching systems sit above the operating system and act as a bridge between object storage and compute frameworks, pulling hot data onto the local NVMe or SSD of each compute node so that reads happen at local speed without physically moving or duplicating the dataset. In one benchmark run (MLPerf Storage v3.0), this kind of layer demonstrated checkpoint write bandwidth of 147.03 GiB/s for a Llama 3 workload, scaling close to linearly as more workers were added, all while the data of record stayed in Amazon S3. One version of this approach, a write-cache feature, cut write latency down to roughly 4 to 6 milliseconds by decoupling write performance from whatever the backend object store could sustain on its own.

The honest caveat here: a caching layer like this is not a general-purpose file system. It doesn't implement full POSIX semantics, and it carries no durability of its own, relying entirely on the object store underneath for that. Any workload that assumes POSIX atomicity is taking on real correctness risk if it assumes the cache layer provides that.

At the hardware layer, NVIDIA's GPUDirect Storage takes a more surgical approach: it removes the CPU from the data path entirely, letting data move straight from storage into GPU memory instead of routing through CPU and system memory first. Meta's production deployment of this is the clearest public example, with Introl documenting a 3.8x improvement in training speed once the CPU bottleneck that had been capping data loading was removed. NVIDIA's cuObject v1.2.0, released in May 2026 alongside CUDA 13.3, extends that idea to S3-compatible object storage over RDMA. It splits the control plane, ordinary HTTP S3 requests, from the data plane, which pushes and pulls data directly into GPU memory using Dynamically Connected RDMA, skipping POSIX scratch staging altogether.

Cloud providers have also built caching directly into their managed storage products. AWS S3 Files, which went GA in April 2026, is one of several managed cloud caching services alongside AWS FSx for Lustre and GCP Anywhere Cache, and it runs a two-tier design internally: small, frequently accessed files get served from a high-performance layer at sub-millisecond to single-digit-millisecond latency, while anything 128 KB or larger gets served automatically straight from S3. Services like this are convenient and fast, but they solve the latency problem within one cloud provider's fence line. They don't address data gravity for teams whose data is spread across more than one cloud, or sitting on-premises.

That's the gap a cloud-agnostic caching layer is built to close. Teams running workloads across S3, GCS, R2, and Azure Blob get the most value from a caching layer that mounts every one of those backends the same way, letting the NVMe tier sit between compute and whichever bucket the data happens to live in, with no migration step and no custom ETL pipeline required. It's the storage equivalent of a universal remote: one interface, regardless of which brand of television is actually plugged in.

Cross-region training: the hardest case for predictive NVMe caching

If S3's latency problem has a boss level, this is it. Cross-region training pits compute in one AWS region against a dataset sitting in another, widening S3's latency gap to its maximum: object storage's native slowness, plus tens of milliseconds of round-trip network latency on every single cache miss. In that setup, a low hit rate makes the whole thing unusable.

There's a documented architecture that shows this working.

This is the scenario that makes the abstract idea of "predictive NVMe caching" concrete. The cache actively learns access patterns, predicting which blocks the training job will ask for next, and prefetches them across a wide-area network link before they're needed, rather than just catching hot reads off a dataset sitting conveniently next door. What would otherwise be a permanent data-gravity tax, the cost of your compute and your data living in different zip codes, turns into a one-time warmup cost paid early in the run and never paid again.

KV cache offload to object storage: where the NVMe tier's role shifts

Training and inference put very different demands on this same caching pattern, and the KV cache is where that difference becomes obvious. During training, the NVMe tier mostly exists to keep a hungry GPU fed from a dataset that gets read over and over across many epochs. During inference, particularly with long-context or multi-turn conversations, what's being cached is the KV cache: the running memory of attention keys and values a model builds up as it processes a conversation, so it doesn't have to recompute that context from scratch on every new token.

That KV cache can get large fast, especially with long conversations or large context windows, and keeping all of it in GPU memory indefinitely isn't practical. Offloading it to object storage solves the capacity problem, but it reintroduces the exact latency problem this entire piece has been describing. Interactive sessions run on a budget measured in a few hundred milliseconds total. If pulling that KV cache back from object storage burns through most of that budget before the model generates a single token, the fact that the GPU itself is fast becomes irrelevant.

This is where the NVMe tier's job quietly changes shape. In training, it's optimized for sustained throughput, constantly feeding a steady stream of batches to a GPU working through the same dataset across many epochs. In inference, it's optimized for tail latency on a much smaller, much more latency-sensitive payload, getting one specific conversation's context back fast enough that the user never notices it was sitting in object storage. The same architectural pattern applies, with the same NVMe tier sitting between S3 and compute. Different job, different clock, same underlying truth: object storage was never built to be fast, and the cache is what makes the gap invisible to whoever's waiting on the other end.