Lustre vs S3 for High-Throughput Workloads
Lustre handles the I/O patterns that make S3 a bottleneck for training and checkpointing at scale.
S3's value is real. Massive durability. No capacity ceiling. No pre-provisioning. No migration headaches. S3 Standard stores data across multiple availability zones, so your source of truth doesn't disappear because something in one data center caught fire. For storing the full training corpus, model artifacts, and long-term checkpoint archives, S3 is excellent.
The problem shows up when you try to use it like a file system.
Every S3 read is an HTTP API call. That call carries authentication overhead, network round-trip cost, and replication pipeline latency baked into every single request. S3 Standard's time-to-first-byte is fine for batch data movement and completely painful for tight training loops. Your GPUs want a fire hydrant; S3 is handing them a garden hose. S3 Express One Zone brings latency down to single-digit milliseconds, which helps, but you're trading cross-zone durability to get there.
There are also structural limits that don't bend regardless of which S3 tier you use:
No atomic rename. ETL pipelines have to implement their own staging logic and application-level locking.
No file locking or shared cache. Concurrent writers to the same object create correctness problems.
Throughput scales by multiplying prefixes. High-concurrency workloads hit rate limits that require architectural workarounds.
Small files are brutal. ML datasets often live as millions of individual samples, which means millions of API calls per epoch, each carrying full latency cost.
For storing data at rest, these limits are mostly invisible; for serving data live into a training loop, they become the bottleneck. That distinction is the whole ballgame.
Lustre Was Built Specifically for the I/O That Breaks Everything Else
Lustre is a distributed parallel file system. The core idea is not complicated: many clients read and write simultaneously across many storage targets, and aggregate bandwidth scales with the cluster. Add more storage targets, get more throughput. That's the whole thing.
It gives you full POSIX semantics. Atomic rename, file locking, memory-mapped files, hard links. Your existing HPC and ML code runs against it without modification. No rewriting your data loader to work around object-store quirks.
Metadata and data are handled separately. Metadata servers manage namespace operations like directory listings, file creation, and renames. Object storage targets handle bulk I/O. That separation is what lets Lustre keep pace with GPU clusters at scale without collapsing under its own overhead.
Production deployments show how seriously teams take this. FSx for Lustre, AWS's managed Lustre offering, delivers sub-millisecond latencies and throughput in the hundreds of gigabytes per second range. The Lincoln Laboratory Supercomputing Center's TX-GAIN cluster, a 2025 build that appears on the Top500 list, mounts central Lustre parallel storage arrays. That's not a pilot; that's how production GPU clusters are actually wired.
Checkpointing is where Lustre earns its keep most clearly. Writing a checkpoint for a very large model means pushing a massive amount of data in a very short window. Lustre's striped I/O spreads those writes across many targets simultaneously; cloud object storage can sustain only a fraction of that throughput under the same burst. Delays during checkpointing translate directly into idle accelerator time, and idle accelerators are the most expensive failure mode in modern AI infrastructure.
Now, the cost. FSx for Lustre runs roughly $0.17 to $0.30 per GB per month depending on throughput tier; S3 Standard runs roughly $0.023 per GB per month. That's a cost differential of five times or more, often much more. That gap is the central reason you don't just run everything on Lustre and call it a day.
Training, Checkpointing, and Inference Don't Just Have Different Speeds. They Have Different Problems.
Most architectural mistakes in ML infrastructure trace back to treating these three workloads as versions of the same problem. They're not.
Training is a sustained throughput problem. The training loop reads the corpus repeatedly. Every epoch is another full pass at multi-gigabyte-per-second aggregate rates across the cluster. The storage fabric has to deliver that throughput continuously, not at peak benchmark numbers. Sustained. Any throughput cliff causes GPU stalls. Small files make this worse because a dataset stored as millions of individual samples generates millions of I/O operations per epoch, and every one carries its own overhead.
Checkpointing is a burst write problem with a hard deadline. Checkpoints need to complete in a narrow window because every minute flushing a checkpoint is a minute the cluster isn't training. A tiered approach handles this well: fast checkpoints frequently to Lustre, periodic flushes to S3 for durability. The frequency difference between those two tiers is what makes the strategy affordable.
Inference and RAG are latency and concurrency problems. A RAG query retrieves a small number of objects in milliseconds; total throughput numbers are modest. But tail-latency and concurrency requirements are aggressive. This profile doesn't need Lustre-scale throughput, but it does expose S3 Standard's latency floor as a real problem for real-time applications. S3 Express One Zone or an NVMe cache layer fits this workload better than either extreme.
No single storage tier serves all three profiles optimally. That's not a flaw in the design. It's just the nature of the workload mix, and you have to plan around it.
Lustre as Hot Tier, S3 as System of Record — How the Two Systems Work Together in Production
The pattern that actually works isn't choosing between Lustre and S3. It's using them together, deliberately.
S3 stays the source of truth. Full dataset, all model artifacts, long-term checkpoint archive. Everything lives there. Lustre sits in front as the active working set, meaning the data the cluster is training on right now. A rough rule of thumb from practitioners: a small fraction of total data needs to be in the fast tier at any given time. The rest waits in S3 until it's needed.
AWS's FSx for Lustre implements this natively through Data Repository Associations. A DRA lets FSx for Lustre present an S3 bucket as part of the POSIX namespace. Files appear in the filesystem without requiring a full pre-loaded copy. Data is fetched on first access, then evicted once it's synced back to S3 and aged out, freeing fast-tier capacity for hotter datasets. The training job sees a regular filesystem and never touches prefixes or API calls directly.
Outside AWS, Lustre has supported Hierarchical Storage Management natively since version 2.5. Copytools exist for S3, Google Drive, HPSS, and other targets. The same policy-driven tiering applies. Hot data on Lustre, cold data on object storage.
The checkpointing workflow makes the pattern concrete. Fast-tier checkpoints written frequently to Lustre handle the most common failure modes with minimal overhead. Periodic flushes to S3 provide the durable, offsite copy. Right-sizing the Lustre deployment to cover only the active working set, rather than the full dataset, can dramatically reduce fast-tier costs, with everything else absorbed at S3 pricing.
The Seam Between Lustre and S3 Is Where Things Get Weird Fast
The places where the two systems hand off to each other don't make it into most architecture diagrams. They should, because that's where the surprises live.
Data movement is its own workload. Staging data from S3 into Lustre before a training run takes time, and that time is GPU idle time if the pipeline isn't managed carefully. Teams that treat data movement as a one-time setup step figure out pretty quickly that it's actually an ongoing operational cost, especially when datasets change between runs.
Consistency at the boundary is not free. S3's consistency model can create subtle correctness issues when Lustre caches a stale version of an object. ETL pipelines that assume POSIX atomics on the S3 side have to implement their own coordination. Staging prefixes and application-level locking aren't optional guardrails; they're the price of using the two systems together without introducing silent data errors.
Checkpoint flushes back up in ways that compound. When a large checkpoint is being flushed from Lustre to S3, S3's throughput ceiling can create backpressure that delays the next checkpoint cycle. If the flush isn't complete before the next fast-tier checkpoint is due, the fast tier fills. The job stalls waiting for a flush to finish, and nobody notices until training has been sitting idle for longer than anyone wants to admit. Sizing the flush window wrong is exactly how you find this out; it's not a fun way to spend a morning.
Metadata doesn't tier gracefully. S3 has no native metadata namespace equivalent to what Lustre provides. Listing a large S3 prefix is expensive and slow compared to a directory listing on a POSIX filesystem. Workloads that rely on fast metadata access, like shuffled dataset iteration or checkpoint discovery, hit unexpected latency even when the data itself is sitting right there in the fast tier.
Designing around these seams isn't optional work you get to if there's time. It's load-bearing. The performance gains the architecture promises don't materialize unless the handoff between systems is treated as carefully as the systems themselves.
The Storage Decision Framework Nobody Frames Honestly Enough
Most teams at scale don't need to choose between S3 and Lustre. They need to understand where each one fits and design the handoff between them as carefully as they designed the training loop itself. Here's how to think about it.
Use S3 alone when:
Access is infrequent and latency tolerance is high (archival, artifact storage, dataset staging)
Your data loaders already stream from object storage and can absorb the latency floor
Storage throughput genuinely isn't your current bottleneck
Use Lustre (or a parallel file system) when:
You're running multi-GPU or multi-node training where aggregate throughput demand exceeds what object storage can reliably sustain
You have a hard checkpoint time budget and object storage throughput limits will be hit in that window
Your HPC or ML code relies on POSIX semantics and rewriting around object-store constraints isn't worth the engineering cost
Use the tiered architecture when:
The full dataset won't fit cost-effectively in the fast tier, but the active working set needs low-latency serving
You have mixed workloads (training, checkpointing, inference) with genuinely different storage personalities
GPU idle time is measurably attributable to storage throughput
For inference and RAG specifically, consider NVMe caching before you reach for a full parallel file system. The problem in inference workloads is usually latency and concurrency rather than aggregate throughput; NVMe caches attached to compute can close most of the gap without a full Lustre deployment. Archil takes this approach, caching S3 data on NVMe attached to compute and presenting it as a POSIX filesystem with sub-millisecond latency. It removes per-request HTTP overhead without requiring a separate storage tier or changes to existing code. It won't replace Lustre for large-scale training throughput; for inference, though, it hits the right tradeoffs without the operational overhead.
The teams that get this right didn't find a better product. They figured out what their workload actually needed before anything was deployed, and then they built the data movement pipeline between tiers like it was part of the system. Because it is.
