Shared SSD Caching Layers for S3-Backed Compute Fleets

Object storage is slow by design. That's not a knock on S3; it's just physics plus bureaucracy, since every read pays a toll for HTTP overhead, authentication checks, and replication across availability zones. That toll runs 10 to 100 milliseconds per request, regardless of whether the object is 4KB or 4GB. NVMe and networked block storage answer reads in under a millisecond. Do the math and you're looking at two or three orders of magnitude, which is a fancy way of saying your GPU cluster spends a lot of its life staring at a loading spinner.

That gap raises idle GPU time, which actually costs money. A team running training jobs straight against S3 can see GPU utilization starved by I/O wait, with schedules stretching far beyond what the hardware alone would suggest. The fix wasn't a faster GPU or a bigger cluster. It was putting a shared NVMe-backed cache in front of S3, which pushed utilization to 95%. Same GPUs, same model, same data. The only variable that moved was how fast the bytes could reach the chip, which tells you where the real bottleneck was hiding the whole time.

The shared SSD caching layer between S3 and compute

Without the branding, it is a pool of NVMe or SSD drives, shared across every node in the fleet, sitting between the GPUs and S3. Reads hit the cache first. A hit comes back at local-disk speed. A miss sends the cache off to fetch from S3, store a copy, and hand it over, so the next request for that same object never makes the trip again.

"Shared" "Shared" is the word doing the heavy lifting there, and it's what most teams get wrong when they build this themselves. A per-node local cache means each machine builds its own little pile of hot data. That leads to duplication, staleness, and twelve nodes independently hammering S3 for the same file because none of them knew the other eleven already had it. A shared layer fixes that outright: a cache hit on node one is instantly visible to node fifty. No duplication, no drift, no thundering herd of requests hitting the bucket at once.

S3 remains the source of truth here, which makes the whole design safe to mess with. The cache isn't a durable second copy of anything, it's an acceleration tier you can flush, resize, or wipe entirely without losing a single byte, because the real data sits safely back in the bucket the whole time. The standard production layout for serious training and inference work now runs three tiers, with S3 for warm and cold storage, a shared NVMe layer for hot data, and GPU memory on top. PyTorch FSDP, DeepSpeed, and Ray Train are all built around that same shape, showing that independent convergence on one architecture, not a fad.

Sizing and populating the caching layer: working set analysis, eviction, and warm-up

Sizing the cache to the dataset is the wrong instinct, and it's the mistake that gets teams to overprovision expensive NVMe for no reason. Size it to the working set instead: the slice of data actually touched during one epoch or one inference window. A dataset measured in petabytes might only need a few terabytes of hot cache, because training access patterns are repetitive by nature: the same shards, over and over, epoch after epoch.

Warm-up latency is the tax nobody budgets for. The first epoch is always slower, because every read is a cache miss making the full round trip to S3 and back. The practical answer is to stage datasets ahead of the job kicking off, specifically so the first pass through the data isn't the slowest one.

One benchmark configuration shows how lean this can get: nodes each carrying eight NVMe SSDs totaling 6TB of cache, feeding a Ray cluster. That's a modest pile of hardware to keep a distributed job fed, and it works precisely because the cache doesn't need to hold everything, just whatever's actively in play. At the other extreme, one large LLM training company built a 360TB cache pool out of idle infrastructure, pushing 70GB/s of throughput. Same idea, scaled to a working set that's genuinely enormous. Either way, the sizing question is about the shape of the access pattern, never the size of the dataset sitting in the bucket.

Implementations: how CoreWeave LOTA and other caching layers approach the same problem differently

Everyone agrees on the problem. Nobody agrees on where the caching layer should physically live, and that disagreement is where the real tradeoffs hide.

CoreWeave's answer, announced October 16, 2025, is LOTA (Local Object Transport Accelerator), and it throws out the separate-fleet idea. LOTA runs as a proxy embedded directly on the GPU nodes, presenting a local cache across the node's own disks that acts as an S3 endpoint from the outside. It moves up to 7GB/s and claims eleven nines of durability. Stage a dataset once and it's reachable anywhere with no egress charges and no per-request fees, watched over through Prometheus and Grafana. One independent assessment put the cost more than 75% below alternative approaches. It only exists inside CoreWeave's own infrastructure. Baking the cache into the node cuts down on moving parts, but nobody's spinning up LOTA on their own hardware in a different data center.

Some tools take the opposite bet: a dedicated caching layer that sits apart from compute, treating the cache as its own service rather than something welded to the GPU node. That separation costs you a network hop, but it buys portability, since the same cache fleet can serve any cluster that can reach it, not just the one it happens to be bolted to. A third path exposes the same data through a file interface and an object interface simultaneously, which erases the caching hop as a separate thing to manage. Three different architectures, one identical goal: put fast, addressable storage between the GPU and the slow bucket, and make it disappear from the code doing the training.

AWS S3 Express One Zone as a managed alternative to a self-operated cache

AWS built S3 Express One Zone for exactly this pain; treat it as the managed middle option rather than a replacement for a real cache. It promises consistent single-digit millisecond latency, roughly 10x faster than standard S3, at request costs up to 50% lower.

Standard S3 runs 10 to 100 milliseconds per request. Express One Zone gets that down to single-digit milliseconds. That's a genuine improvement, and for plenty of workloads it's enough. But it's playing a different sport than an NVMe cache, which is sub-millisecond, next to the GPU rather than a network call away. Express One Zone is a faster bucket, not a substitute for local flash.

It also has a hard ceiling. Each directory bucket caps out at 200,000 reads and 200,000 writes per second. A small or mid-size job never bumps into that number. A large distributed training run hammering one bucket from thousands of workers will, so check fleet size against that ceiling before betting the whole pipeline on it. One pattern that's caught on is syncing objects from a standard S3 bucket into an Express One Zone directory bucket ahead of a job, which amounts to a managed warm-up routine instead of hand-rolled prefetch code.

Caching layer capabilities under a POSIX filesystem over object storage

S3 speaks a narrow language limited to get, put, and delete. That's the whole vocabulary. No append, no seek, no rename, no file locking, no sparse files. Any training code, checkpoint library, or data loader written assuming a real filesystem simply can't talk to S3 without a rewrite, and nobody has time for rewrites mid-project.

Atomic rename lets a checkpoint write finish safely without corrupting the last good checkpoint if the job dies mid-write. File locking (flock, fcntl) stops two processes from stepping on the same file during concurrent writes. Memory-mapped files (mmap) let model weights load without a full copy into working memory first. fsync gives a hard durability guarantee before the next step proceeds, and hard links or symlinks let teams version datasets without physically duplicating terabytes.

None of that exists in raw S3. Putting a POSIX-compatible cache in front of it makes all of it work unmodified, with no SDK swap, no adapter code, and no maintenance headache every time a library bumps its S3 client version. It also means a person, or an AI agent trained on command-line habits, can poke at the dataset directly with ls, grep, find, and cat, the same way they'd explore any folder on a laptop instead of learning a bespoke API just to look at a filename.

Operational considerations: multi-tenancy, consistency, observability, and cache invalidation

A cache shared across multiple teams is a shared resource, and shared resources get fought over eventually. One job's dataset can flood the cache and evict another job's working set. That failure mode has a name, cache pollution, and the result is everyone's job slows down even though only one of them is actually misbehaving. Isolation between tenants is the baseline requirement for any cache meant to serve more than one team without turning into a blame game.

Consistency and invalidation questions sit right behind that: when the object in S3 changes, does the cache find out, and how fast does that news travel? Observability matters just as much. A cache that silently misses far more often than expected is worse than no cache at all, since it adds a network hop without delivering the speed anyone's counting on, and nobody notices until the training run is mysteriously slow again.

The pattern itself, a shared NVMe tier between object storage and compute, has become common enough to ship as a managed product instead of something every team hand-builds. Managed products now exist that mount S3 buckets as a native POSIX filesystem with an integrated NVMe cache, serving reads at high speed on a hit and falling back to S3 on a miss, with no code changes and no data migration required. Archil, for instance, is a serverless filesystem layer that mounts existing S3 or GCS buckets directly, giving AI workloads zero-copy access without moving the underlying data. Self-managed cache fleet, node-embedded proxy, managed hot-tier bucket, or mounted filesystem layer: the math never changes because S3 is durable and cheap, NVMe is fast and finite, and the caching layer's entire job is making the GPU forget it was ever waiting on anything.