NVMe vs SSD Caching for S3 Read Workloads
NVMe caches deliver 10x faster IOPS and lower latency than SATA for S3 workloads that need it.
S3 is slow for a simple reason: it is effectively a massive pool of spinning disks sitting behind an HTTP API, and every request has to travel over HTTPS, pass through an API gateway, and pull data back from a distributed system that might span several physical locations. None of that is a bug: it is just what object storage is built to do, and the latency floor comes from network and distributed-system overhead, not from some fixable setting buried in a config file.
The numbers are blunt. S3 object storage latency is measured in tens of milliseconds per request, and each of those requests yields less than 100 MB/s. That ceiling traces back to the physical nature of spinning disks, and no amount of clever tuning on the operator's side moves it. Solid-state drives, meanwhile, run at access latencies that are orders of magnitude faster than S3, with millions of IOPS per device, and they can read in chunks as small as 4 KB. Because of that gap between S3 and SSDs, almost every serious piece of software built on top of S3 is, in some sense, quietly designed around S3's limitations.
The fix that the industry settled on is structural rather than clever: put a local caching layer between S3 and the compute that needs the data. A cache absorbs repeated reads, so you pay the expensive S3 round-trip once, on the first cold miss, instead of on every single access. That solves the slow round-trip to S3. It does not solve the next one, because the cache tier itself is built out of hardware, and the gap between the two most common choices, NVMe and SATA SSD, turns out to be surprisingly wide.
How far apart NVMe and SATA SSD are as cache hardware
The distance between NVMe and SATA SSD isn't a rounding error you can shrug off, appearing in throughput, in IOPS, and in latency, which covers basically every dimension a read-heavy workload cares about.
The reason traces back to the interface, not just the flash chips inside. NVMe talks directly to the CPU over PCIe and skips the old SCSI command layer that SATA still depends on, and it supports tens of thousands of queues, each carrying tens of thousands of commands, while SATA is stuck with a single queue holding 32 commands. Think of SATA as a single-lane tollbooth and NVMe as a stadium parking lot with thousands of exits.
Benchmarks at the 3.84 TB drive tier make the comparison concrete. The Micron 5400 PRO, a representative SATA drive, delivers 540 MB/s sequential read, 95,000 random-read IOPS at QD32, and 150–300 µs average read latency. NVMe drives in the same class run roughly 10 times faster on IOPS and 2 to 5 times lower on latency at that same queue depth.
Set either of those numbers against S3's tens-of-milliseconds first-byte latency, and both NVMe and SATA look like a different category of hardware, fast enough in principle to hide the S3 penalty for data that's already been cached. But the gap between NVMe and SATA still matters on its own terms, because it decides whether a demanding workload ends up bottlenecked on the cache drive itself rather than on S3. And the cost story doesn't tilt as hard toward SATA as the IOPS numbers might suggest: at the 3.84 TB tier, NVMe (the Samsung PM9A3) costs somewhat more per GB than SATA (the Micron 5400 PRO), but the gap is a lot narrower than most teams expect given how much more performance NVMe delivers.
The workload characteristics that determine which cache tier wins
You don't pick a cache tier by declaring one drive the overall winner. It comes down to three properties of the workload sitting on top of it: how much latency it can tolerate, how often it revisits the same data, and how many concurrent reads it throws at the cache at once.
Latency budget is the first filter. A workload that can live with sub-millisecond reads but doesn't need sub-100-microsecond reads, think sequential analytics jobs or batch inference, gets everything it needs from SATA's 150 to 300 microsecond latency, and paying extra for NVMe buys nothing in that case. Flip the requirement to tens of microseconds, the territory of OLTP-style random lookups, real-time inference serving, or KV-cache retrieval during large language model inference, and SATA turns into the bottleneck. NVMe stops being optional.
Access pattern is the second filter, and it's the one people skip past most often. NVMe earns its recommendation for AI training data because the GPU pipeline can't wait on storage, but that only holds if you revisit the hot dataset often enough for the cache to build a high hit rate. SSD caching of any kind, NVMe or SATA, is built for repeated reads of the same high-latency data. An epoch-shuffled training dataset, where each sample is unlikely to repeat within a run, defeats that design regardless of which drive sits underneath it, because the cache hit rate stays near zero no matter how fast the drive is.
Queue depth is the third filter, and it's where the architecture gap from the previous section turns into a practical wall. A single command queue is fine for light, mostly sequential work, but it becomes a hard ceiling the moment parallelism increases: multiple GPUs pulling data at once, a pool of inference threads, or a distributed training job firing off thousands of small concurrent reads. NVMe's 65,535-queue design means it scales as concurrency rises instead of forcing requests to line up single-file, and its IOPS advantage over SATA actually grows wider as queue depth increases, which is exactly the condition that high-throughput AI training and multi-tenant inference serving create.
Where NVMe caching effectively erases S3's performance penalty
For a workload that combines deep queue depth, high concurrency, and repeated access to a bounded hot dataset, local NVMe caching can reduce effective read latency from S3's tens of milliseconds down to tens of microseconds on a cache hit. That's a two-order-of-magnitude jump, and it doesn't just make things faster. It changes what kind of computation is realistic to run at all.
AI training is the clearest case. When storage can't keep pace, GPU utilization collapses, because the bottleneck becomes data throughput from storage into GPU memory rather than the GPU's own compute capacity, and training jobs routinely run at a fraction of the GPU utilization they could otherwise hit. The fix isn't fetching data on demand during the run. You pre-stage the hot dataset onto local NVMe before the job starts, so the data moves once instead of competing with training traffic for the entire run. Netflix's production tiering shows the same logic at a much larger scale: NVMe holds a small slice of total storage capacity but absorbs most of the request volume, because that's exactly the job NVMe caching is built for, serving hot data that keeps getting hit.
Large language model inference tells a similar story through its KV-cache. LMCache, deployed in production at enterprise scale by CoreWeave for Cohere, intercepts prefix tokens during prefill and writes the resulting KV tensors through a layered hierarchy: CPU memory, then local NVMe, then S3-compatible object storage, with NVMe serving cache hits at microsecond latency instead of forcing every request back out to S3. SATA would have struggled here specifically because the inference latency budget is tight enough that NVMe's 60 to 80 microseconds versus SATA's 150 to 300 microseconds compounds across a high volume of requests.
ClickHouse Cloud built a distributed cache for cloud object storage because per-node SSD caches kept hot data siloed on whichever node first fetched it, so every other node had to re-fetch the same data from object storage all over again. Their measured latency stack puts object storage at hundreds of milliseconds in the worst percentiles, local SSD around 1 millisecond, and memory at 250 nanoseconds, and query time from cold to warm drops substantially once the cache fills in. A shared distributed cache can beat isolated per-node NVMe caches when many compute nodes all need the same hot data, because it avoids making every node re-fetch the same bytes from S3.
GPUDirect Storage closes out the case for NVMe. Meta adopted GPUDirect Storage and removed a CPU bottleneck that had been limiting data loading, so training data now streams straight to GPUs at far higher throughput, after PyTorch jobs had burned a meaningful chunk of compute time just waiting on data. S3 doesn't support GPUDirect Storage on its own. Data has to land on local NVMe first, because that's what supplies the GDS-compatible local read path, so if you build around this architecture, NVMe is a requirement, not a nice-to-have. A newer option, the cuObject library, now generally available, tries to stream data via RDMA directly from S3 to GPU VRAM using an HTTP header mechanism, but it depends on InfiniBand or RoCE networking with GPUDirect RDMA-capable NICs, so it fills a narrow niche rather than replacing local NVMe staging across the board.
Where SATA SSD caching breaks down
SATA SSD earns its place when shallow queue depth, a sub-millisecond rather than tens-of-microseconds latency budget, and cost per GB as a real constraint rather than an afterthought all line up. That's not a narrow edge case. It describes a large and ordinary slice of real workloads.
Sequential analytics jobs read large objects in order, so they fit comfortably within SATA's 540 MB/s sequential throughput, because the bottleneck in those jobs tends to be the S3 fetch itself, not anything that happens after the data lands locally. Backup and restore staging fits too: these are write-once, read-rarely patterns with no real queue depth demands, and SATA gets explicitly recommended for backup targets because cost per TB affects total storage spend more than speed does there. Lightly concurrent workloads with modest IOPS needs also do fine on SATA. Its 95,000 IOPS is more than sufficient for workloads that do not saturate it, and its 150 to 300 microsecond read latency still beats S3's tens-of-milliseconds baseline by orders of magnitude.
The cracks appear under concurrency, marking where SATA SSD breaks down. A single command queue serializes I/O once parallelism gets deep enough, so a multi-GPU training node or a multi-tenant inference server firing off hundreds of concurrent small reads will exhaust that queue, and the queuing delay this creates eats into whatever value the cache was supposed to add. AI training with epoch-shuffled datasets can produce near-zero cache hit rates no matter which drive sits underneath, since that access pattern defeats caching. But for training jobs that do have locality in their data, SATA's IOPS ceiling against NVMe's becomes the actual bottleneck, not S3. As a rough yardstick, SATA SSD delivers tens of thousands of random read IOPS while local NVMe delivers hundreds of thousands, and this gap grows once queue depth climbs past what SATA can serve concurrently.
These middle options genuinely help in some cases and genuinely fall short in others, and the workload properties from earlier in this piece, latency budget, access pattern, and queue depth, decide which bucket a given job falls into.
Faster S3 tiers close part of the gap between object storage and local disk, but they operate within the same architectural family as standard S3: requests still travel over HTTPS, through a gateway, to a system that wasn't built to behave like a local NVMe drive sitting a few inches from the CPU. That architecture puts a floor under how far latency can drop, a floor that has nothing to do with configuration and everything to do with what object storage fundamentally is. Managed cache services built by cloud providers or data platforms solve a real problem, namely saving teams from having to provision and babysit their own cache hardware, but they inherit the same tradeoffs described throughout this piece: a shared, distributed cache can beat isolated per-node caches when many compute nodes need the same hot data, exactly the pattern ClickHouse Cloud hit when per-node SSD caches kept forcing redundant re-fetches from object storage across nodes. That's a genuine advantage for workloads with wide fan-out across many nodes. But it's no universal substitute for local NVMe, especially for workloads like GPU training and inference serving, where you need to keep data close enough to the compute that it never has to leave the box.
The practical framework holds regardless of which packaging a team picks, whether that's raw S3, a faster S3 tier, a managed caching layer, or a self-managed NVMe or SATA tier. First check the workload's latency budget, then confirm whether its access pattern reuses data often enough for caching to help at all, and then size the queue depth the cache tier needs to sustain under real concurrency. If you skip that sequence and pick hardware or a managed service on specs alone, you end up with a fast drive sitting idle behind a slow workload, or a cheap drive buckling under one it was never built to carry.
