Caching Layers in Front of S3

Cloud infrastructure separates compute from storage on purpose. You scale each one independently, pay for what you use, and life is good. The problem shows up when you put a powerful GPU cluster on one side and S3 on the other. The network in between becomes the bottleneck. The GPUs aren't slow. The data pipeline is. Think of it like a Formula 1 car stuck behind a school bus — the engine isn't the problem.

Meta found that 56% of GPU cycles were stalling while waiting for training data. More than half. They ended up building a dedicated data preprocessing service just to stop that from happening. That's not a minor tuning exercise. That's a serious engineering investment made because the storage layer was costing them more than the hardware itself.

Some numbers to put that in context:

Well-optimized training pipelines can sustain 85 to 95% GPU utilization during active training.
Large language models can require more than 10 GB/s of sustained read throughput to run properly.
Without a caching layer, most teams land well below both of those numbers.

Here's the part that actually stings. When you buy more powerful GPUs to fix a performance problem, but the problem is your storage pipeline, you haven't made things faster. You've made idle time more expensive. The hardware is better. The bottleneck is the same. You've paid more for the privilege of watching it not work.

S3 is fine for raw datasets, archives, and checkpoints sitting at rest. The failure mode shows up during active training throughput, where S3's latency, measured in tens of milliseconds per request, crosses from annoying to genuinely costly.

What a Caching Layer Actually Does Between the Application and S3

A cache sits in front of S3, intercepts read requests, and serves data locally if it already has a copy. If it doesn't, it fetches from S3, serves the data, and stores a copy for next time. First request is a miss. Every repeat request after that is a hit. That's it. You could say the cache is like a well-read librarian — it remembers what everyone keeps asking for and stops making the trip to the archive every single time.

Most production setups use three tiers:

Memory layer. Fastest path. Sub-millisecond lookups. Limited capacity. Holds the hottest fraction of your working set.
Disk layer (typically NVMe). Much larger than RAM. Far lower latency than S3. Catches warm data that falls out of memory.
Object store (S3). Source of truth. Cold data and durability live here. The cache never replaces it.

What gets promoted to memory, what gets pushed to disk, and what gets evicted is controlled by the eviction policy. LRU (least recently used), FIFO (first in, first out), and newer algorithms like w-TinyLFU and SIEVE each make different assumptions about your access patterns. The right choice matters more as your working set grows.

Writes can flow two ways:

Write-through. The cache writes to S3 and waits for confirmation before telling the application it's done. Safer. Slower.
Write-back. The cache confirms to the application immediately, then flushes to S3 in the background. Faster, but there's a window where a cache failure could mean data loss.

Most caches also do lazy loading. Data only gets fetched from S3 on first access, no bulk prefetch required. Capacity stays proportional to what you actually use, not what you store.

One practical example: S3 Files uses a size threshold to decide how to handle different objects. Files below 128 KB get served from cache. Larger files get streamed directly from S3, where sequential throughput matters more than low-latency lookup.

The benefit shows up in three places: lower latency, higher throughput, and fewer S3 API calls. That last one shows up directly on your monthly bill, which always gets people's attention.

Where the Cache Sits Determines What Problem It Solves

Cache placement is not a detail. It determines what problem you're actually solving.

Colocated with compute (same host or same AZ). Lowest possible latency. Highest throughput. Right answer when data needs to move to a GPU fast.
Distributed caching layer (between your cluster and S3). Scales horizontally. Shared across multiple compute nodes. Good for multi-tenant environments or parallel jobs.
Edge or CDN (geographically distributed). Minimizes round-trip for users around the world. Right tool for content delivery. Wrong tool for a training cluster.
Managed file system with S3 backend. Exposes object storage as a POSIX filesystem. Lets existing code run without modification.

AWS makes the colocality rule explicit with S3 Express One Zone. The performance gains are specifically tied to having compute and storage in the same availability zone. Move them apart and the benefit shrinks fast.

Hot working set with random small reads? You want the cache close to the GPU. Sequential streaming of large files? The bottleneck is throughput, not latency, and placement matters less than bandwidth. Global users reading the same content? Edge caching is exactly right. Getting this wrong means you've added complexity and cost without fixing anything.

AWS-Native Options: Express One Zone, ElastiCache, FSx for Lustre, and CloudFront

S3 Express One Zone

Express One Zone launched in late 2023 and is the closest thing AWS has to an S3 tier designed specifically for performance. It's not a cache in the traditional sense. It's a high-performance storage class that behaves like one in practice.

Key numbers:

10x faster than S3 Standard.
Up to 2 million reads and 200,000 writes per second.
Single-digit millisecond request latency.

After AWS cut prices in 2025 (85% on GET costs, 55% on PUT, 31% on storage), benchmarks showed roughly 3x better latency than standard S3 at around 15% higher total cost. SageMaker training jobs colocated with Express One Zone storage have seen 40 to 60% shorter training times from eliminated I/O wait.

The trade-off is real: single-AZ means lower durability than S3 Standard. This is fine as a hot tier. It should not be your only copy.

FSx for Lustre

Lustre is the filesystem that HPC clusters have been running on for years. FSx for Lustre is the managed AWS version. It links natively to S3, lazy-loads objects on first access, and serves subsequent reads from the filesystem. To your application, it looks exactly like a POSIX filesystem.

What makes it worth using in 2025:

Sub-millisecond latency, throughput that scales to terabytes per second, millions of IOPS.
EFA plus NVIDIA GPUDirect Storage support (launched late 2024): all eight GPUs on a node can read shards in parallel directly into memory, bypassing the CPU entirely. Model load times that used to take minutes now take seconds.
Intelligent-Tiering (launched mid-2025): automatically moves data between hot, warm, and archive tiers with an optional SSD read cache in front.

Best fit: teams running GPU-intensive training or HPC workloads on AWS who need a POSIX interface over S3 and don't want to build a custom caching layer from scratch.

ElastiCache

ElastiCache is a managed in-memory cache running Redis or Memcached. The pattern is: your application fetches an object from S3, stores the result in ElastiCache, and subsequent reads bypass S3 entirely. Latency drops. Throughput goes up.

The catch is that it requires explicit application-level integration. ElastiCache doesn't intercept S3 traffic automatically. Your code has to decide what to cache and handle the logic itself. That makes it a good fit for caching specific objects or query results at the application layer, not for raw training data pipelines where you want the caching to be transparent.

CloudFront

CloudFront is AWS's CDN. It caches S3 objects at edge locations globally and can reduce latency for distributed end users down to single-digit milliseconds.

It's the right tool for content delivery, large file distribution, and API acceleration for a global user base. It's the wrong tool for training pipelines, because the consumer on the other end of those reads is a GPU cluster, not a browser. They are genuinely different problems and it's worth being direct about that rather than hedging.

Third-Party Distributed Caching Layers: Alluxio and JuiceFS

Alluxio

Alluxio sits transparently between your application and any object store. S3, GCS, Azure Blob. It exposes both a POSIX interface and an S3-compatible API, so you can mount existing buckets without migrating anything. That portability is the main reason teams reach for it over AWS-native options.

Unlike single-node tools like s3fs, Alluxio is fully distributed and handles metadata and data management across a cluster. Vendor-reported benchmarks from 2024 show throughput ranging from around 2,000 MiB/s on a single thread to more than 8,000 MiB/s with 32 threads. Real-world outcomes cited by Alluxio include sub-millisecond time to first byte, GPU utilization above 90% in ML training workloads, and feature store query times dropping by as much as 80x in some deployments. Test against your own workload before committing to those figures.

The known limitation: Alluxio uses an eventual consistency model. In multi-cluster environments where each cluster's cache may hold a different version of the same file, this causes real problems. Some users describe it as a caching glue layer rather than a proper filesystem. Whether that bothers you depends entirely on whether your workload requires consistency across clusters.

JuiceFS

JuiceFS takes a different architectural approach. File data lives in S3. Metadata lives in a separate database, with Redis, MySQL, TiKV, and SQLite all supported. That separation gives stronger consistency guarantees than Alluxio's caching model.

A few real deployments worth knowing about:

BioMap, an AI life sciences company, chose JuiceFS over both Lustre and Alluxio for managing billions of small files. Storage costs dropped 90% versus their prior architecture.
A self-driving company using JuiceFS in a multi-cloud setup, managing tens of billions of files, hit 400,000 IOPS and nearly 40 GB/s of throughput.
Sequential read performance has been measured at 200% faster than s3fs in independent testing.

The trade-off is that running a separate metadata store adds operational overhead. You're now managing both JuiceFS and a database. For workloads where consistency matters, that's worth it. For simpler workloads, it probably isn't.

Multi-cloud portability, strong consistency requirements, or workloads that span heterogeneous clusters are the situations where a third-party option makes more sense than going AWS-native.

How Write-Back Caching Changes the Checkpointing Problem

Checkpointing during large training runs is one of the more painful interactions between ML workloads and object storage. Checkpoint writes to S3 carry tens of milliseconds of latency. At the frequency that serious training jobs checkpoint, that adds up quickly. At scale, those writes can also trigger S3 throttling on top of everything else.

Write-back caching addresses this directly. The write lands in the cache. The cache immediately confirms to the training process that the write is done. The flush to S3 happens asynchronously in the background. The training job doesn't wait, S3 still gets the durable copy, it just isn't on the critical path anymore.

A couple of numbers worth knowing:

Optimal storage selection alone can speed up checkpointing by close to 2x.
Combined with fast networking (InfiniBand is roughly 10x faster than standard Ethernet), these optimizations together can deliver 6 to 7x end-to-end training speedup.

The durability exposure is real and worth naming plainly. Between the cache write and the S3 flush, if the cache fails, you lose that checkpoint. Teams need to decide whether the latency gain is worth that recovery risk before they deploy, not after something fails. Sometimes synchronous replication to a second cache node before the async flush is the right call. Sometimes the checkpoint frequency is high enough that losing one is acceptable. That decision depends on the specific training run and how much you trust your cache infrastructure.

FSx for Lustre with GPUDirect Storage is the clearest production-grade implementation of this pattern inside AWS. Model load times that used to take minutes drop to seconds.

Cache Invalidation, Consistency, and the Edge Cases That Trip Teams Up

Cache invalidation is the classic hard problem in distributed systems. "There are only two hard things in computer science," the saying goes, "cache invalidation and naming things." Training pipelines get to deal with both, often at the same time.

Stale data is straightforward to describe and annoying to actually hit. A cached object goes stale when the underlying S3 object gets updated. The cache keeps serving the old version until the TTL expires or an invalidation event fires. For static training datasets that never change, this isn't a real problem. For anything that gets written and then read back, it absolutely is.

TTL-based expiration works fine for static data. If your training dataset is fixed and you're just reading it repeatedly, a long TTL is completely fine. The Varnish S3 Shield pattern handles a middle ground: it revalidates content on expiry and only fetches from S3 when content has actually changed, which means fewer unnecessary origin hits without sacrificing freshness.

The metadata consistency problem in Alluxio is a specific version of stale data at scale. In multi-cluster environments, each cluster's local cache may hold a different version of the same file at the same moment. JuiceFS's separate metadata store solves this, and yes, it adds operational overhead. That's the trade.

Atomic rename deserves specific mention because it catches teams off guard. POSIX semantics require atomic rename. Object storage doesn't support it natively. Caching layers that advertise POSIX compatibility have to implement atomic rename themselves, and if they get it wrong, applications that rely on it will silently break in ways that are genuinely difficult to debug. Verify this behavior before you assume it works.

The general rule: the more consistency guarantees a caching layer adds on top of S3, the more complex the solution gets. A simple read cache for a static dataset doesn't need distributed transaction semantics. A checkpoint-and-recover system probably does. Match the complexity of your cache to the consistency requirements of your workload, not the other way around.

Matching the Caching Approach to the Workload

Access pattern.

Random small reads from a hot working set: cache colocated with compute. Low latency matters more than throughput here.
Sequential streaming of large files: throughput-optimized. A distributed layer may serve you better than per-node caching.
Global end users reading the same content: CloudFront. That's exactly what it's built for.

Write behavior.

Read-heavy with infrequent writes: almost any cache works. Focus on hit rate and eviction policy.
Frequent checkpointing: write-back caching cuts idle time. Decide your durability tolerance before you deploy, not after something fails.
High-frequency object updates with readers expecting fresh data: eventual consistency caches will cause problems. You need either a strong-consistency layer or a very short TTL, and a short TTL erodes most of the cache's value anyway.

Consistency requirements.

Static datasets read repeatedly: eventual consistency is fine. Alluxio, CloudFront, or ElastiCache all work without stress.
Multi-cluster environments where different jobs read the same files: JuiceFS's metadata store or FSx for Lustre.
POSIX semantics required by existing code: FSx for Lustre or JuiceFS. Verify atomic rename behavior before you deploy anything.

Operational complexity tolerance.

You want AWS to manage it: S3 Express One Zone, FSx for Lustre, ElastiCache, or CloudFront.
You need multi-cloud portability or have consistency requirements that AWS-native options don't meet: Alluxio or JuiceFS, with the operational overhead that comes with them.

Most applications repeatedly request a small fraction of what they store. The data causing the most latency pain is almost always the same data, requested over and over. A well-placed cache with a reasonable eviction policy captures that working set and makes S3's latency irrelevant for most reads. The goal isn't to replace S3. It's to make sure most requests never have to reach it.