S3 vs EFS Performance for Machine Learning Workloads

Neither S3 nor EFS was designed for GPU training pipelines. That's not a criticism. It's just the starting point for every decision that follows, and skipping past it is how teams end up with storage setups that quietly strangle their accelerators.

S3 is object storage. Built for durability and bulk throughput across massive datasets. It doesn't natively understand file paths the way Linux does. Every object is addressed by a key. Capacity scales horizontally without any provisioning on your end, effectively unbounded, and it's the cheapest per-GB option you'll find on AWS by a wide margin. Think of it as the system of record — data lives there and gets pulled toward compute when needed, the way a library holds books until someone checks them out.

EFS is a managed NFS. It gives you a true POSIX file system interface: standard Linux mount, no SDK, no code changes. Multiple compute instances can read from and write to it simultaneously without any coordination logic on your end. It scales automatically. It also costs meaningfully more per GB than S3, and at ML-relevant dataset sizes, that difference stops being a rounding error pretty quickly.

Here's the distinction that drives everything downstream: S3 is optimized for durability and bulk throughput. EFS is optimized for shared access and POSIX compatibility. Genuinely different design goals, built for genuinely different problems. The fact that neither was designed with GPU training in mind is exactly why so many teams end up paying a lot of money for storage that's actively throttling their accelerators.

How S3 Performs When ML Workloads Are Shaped to Match Its Strengths

S3 can be a high-throughput training data source. Whether it actually gets there depends almost entirely on how you access it, and most teams get this wrong in the same predictable way.

The decisive variable is access pattern. Sequential reads of large shards are high-throughput and latency-tolerant. S3 handles these well. Random reads of millions of small files expose S3's per-request latency. That's where things fall apart, and they fall apart fast. It's like trying to fill a swimming pool one teaspoon at a time — technically possible, practically painful.

The single biggest lever for S3 training throughput, per AWS's own guidance, is consolidating your datasets into shards in the range of hundreds of megabytes to a few gigabytes per file, then reading them sequentially. Skip that step and you'll feel it across every training run. It compounds.

The S3 Connector for PyTorch handles request parallelization, connection reuse, retries, and concurrent listing automatically. You don't have to build any of that yourself. And it matters more than people expect. Checkpoint writes through the S3 Connector are actually faster than writing to EC2 instance storage, which sounds backwards until you understand how much the parallelization is doing under the hood.

S3 Express One Zone is the high-performance tier that's relevant here. It delivers single-digit millisecond latency, which is the meaningful improvement for random-read, small-file workloads. It's most useful when the workload is latency-bound rather than throughput-bound. Inference pipelines are the real target, because per-request latency accumulates into real wall-clock cost at serving scale.

S3 at its best looks like this: a large training dataset, well-sharded, read sequentially, with a properly parallelized client. That pattern gets you near-peak throughput at the lowest per-GB cost of anything we'll cover. The catch is you have to actually set it up that way. The default path gets you none of it.

Where EFS Fits and Where Its Per-Client Ceiling Becomes the Problem

EFS has genuine strengths. It also has a ceiling that shows up at an inconvenient moment if you're not watching for it.

Where EFS actually earns its place:

• Shared POSIX namespace. Many training nodes can mount the same filesystem simultaneously with no coordination code.
• Sub-millisecond first-byte latency on the Standard storage class when data is hot.
• Shared preprocessing pipelines, feature stores, and multi-team notebook environments where POSIX semantics like append, rename, and locking genuinely matter.

The per-client throughput cap is the quiet problem. In Elastic Throughput mode, each mounted client is bounded at a fixed ceiling. That ceiling does not scale with the number of GPUs in your cluster. A large parallel training job with many nodes shares that per-client limit per mount. Teams running at scale hit this before they expect to, and by then they've already paid for the EFS storage. Feels great — like arriving at a buffet only to find out it closed five minutes ago.

EFS can reach high aggregate IOPS with quota increases, but the defaults are far lower. If you haven't explicitly planned for this, you'll find out the hard way at the worst possible time.

A few other things that trip people up: General Purpose mode has the lowest per-operation latency. Max I/O trades latency for higher parallelism and is explicitly flagged by AWS as a legacy option. Use General Purpose. EFS IA and Archive storage classes carry meaningfully higher first-byte latency than Standard, so cold-tiering your training data to save money means you'll feel that latency every epoch. And EFS Standard costs roughly ten times more per GB than S3 Standard. At petabyte-scale datasets, that math becomes prohibitive unless the shared POSIX requirement is genuinely non-negotiable.

EFS is not a bad service. It's a service with a specific job. Large-scale parallel training isn't really it.

FSx for Lustre Sits Between Them and Costs Accordingly

FSx for Lustre does something neither S3 nor EFS does natively: it stripes data across multiple storage servers so a large training cluster with many GPU nodes can read simultaneously at full aggregate bandwidth. That architecture was explicitly designed for this pattern, and it shows.

The headline numbers are sub-millisecond latency and throughput measured in terabytes per second at the high end. It also has native S3 data repository integration. Files are lazily loaded from S3 on first access, then served from the high-performance cache on subsequent reads. That lazy-load pattern maps well to iterative ML training. The first epoch is slower. Every epoch after runs at full cache speed.

The operational reality is less exciting. FSx for Lustre requires pre-provisioned storage paid for upfront. Capacity cannot be reduced after provisioning. Resizing causes a brief availability window. These aren't dealbreakers, but they're the kind of thing nobody mentions until you've already committed.

Per-GB cost is substantially higher than both S3 and EFS Standard. That cost is justified in exactly one scenario: the workload genuinely needs sustained, parallelized, sub-millisecond access across many nodes simultaneously, and GPU idle time is the dominant cost in the system. For HPC-scale training runs measured in hundreds of GPUs and lasting days or weeks, FSx for Lustre is the right answer. For fine-tuning, evaluation, and most inference workloads, it's over-engineered and overpriced.

S3 Files Changes the Cost-vs-POSIX Tradeoff for Read-Heavy Workloads

S3 Files is a newer option that reshapes the math on a specific class of problem. What it does: mounts any S3 bucket as a shared NFS file system on EC2, Lambda, ECS, EKS, and Fargate. Full POSIX semantics. Data stays in S3. It's built on EFS infrastructure under the hood.

The cost structure is layered in a useful way. A per-GB cache charge applies only to hot data. Large sequential reads stream directly from S3 at no S3 Files surcharge. That makes it significantly cheaper than EFS for analytics workloads that are predominantly reading large files. AWS claims meaningful cost reductions compared to cycling data between S3 and a separate file system, and that claim is plausible for teams currently running ETL pipelines just to get S3 data into a mountable form.

For ML specifically, the strongest case is read-heavy analytics and inference serving where the data already lives in S3, or workloads where existing tooling requires a POSIX mount and refactoring isn't worth the engineering time.

Where it doesn't change the equation: write-intensive workloads like checkpoint-heavy training runs still favor purpose-built options. Workloads requiring the lowest possible first-byte latency still favor purpose-built options. S3 Files is a meaningful addition to the toolkit. Whether it helps you depends almost entirely on where your data lives and the read/write ratio of the workload you're running.

Matching Storage to the ML Workload Phase Rather Than the Team's Default

The most common mistake isn't picking the wrong storage service. It's picking one service as a default and applying it to every phase of the ML lifecycle. Each phase has different needs, and treating them the same is how you end up paying FSx prices for data that barely moves.

Dataset staging and preprocessing. S3 is the right home. Elastic, cheap per GB, durable, easily shared across teams and regions. The cost of not sharding here compounds into every subsequent training run.

Large-scale training with many GPUs and long runs. Well-sharded S3 with a parallelized client is the highest-throughput, lowest-cost option when the data is properly formatted. FSx for Lustre justifies its cost when the cluster is large enough that GPU idle time outweighs the storage premium. EFS rarely wins here. The per-client throughput cap and the per-GB cost both work against it at scale.

Checkpointing. S3 with a parallelized connector outperforms even local instance storage for checkpoint writes, per AWS's own benchmarks. Most teams treat checkpointing as an afterthought and then wonder why it's eating into their training window.

Shared preprocessing and feature pipelines across multiple teams. This is where EFS earns its keep. POSIX semantics, shared namespace, concurrent access without coordination code. S3 Files is worth a look here too if the data already lives in S3 and the workload is read-dominated.

Inference serving. Storage latency here is a production SLA concern, not a training optimization. S3 Express One Zone's single-digit millisecond latency targets this directly. EFS Standard's sub-millisecond first-byte on hot data is competitive for model weight serving when the model fits the cache. Inference now accounts for the large majority of AI compute cycles in production. Designing storage architecture for serving as an afterthought is one of the more expensive mistakes a team can make, and it's extremely common.

When a Cloud-Agnostic POSIX Layer Eliminates the Choice Entirely

There's a part of the S3-vs-EFS conversation that rarely comes up: both options assume AWS as both your compute and storage provider. The moment you're running workloads across GCS, Azure Blob, Cloudflare R2, or some combination, you have to rebuild the entire decision tree from scratch for each provider. That's real engineering time that produces zero model improvement.

The data movement tax is the hidden cost nobody budgets for. Every bespoke pipeline that copies data from object storage into a mountable file system for training, then writes results back, is overhead. It's not neutral. It's time, money, and complexity that compounds across every team touching that data.

A cloud-agnostic POSIX layer collapses this problem. One that mounts S3, GCS, R2, or Azure Blob as a real file system without ETL, migration, or code changes means the same tooling, the same mount semantics, and the same access patterns work regardless of which object store holds the data.

Archil is built on this premise. Point it at an existing bucket on any supported provider and mount it as a POSIX filesystem with NVMe-backed caching for sub-millisecond read latency on hot data. Writes are replicated before returning and flushed asynchronously to the bucket. The bucket stays the source of truth. No persistent copy outside the customer's account, and access is revocable at any time.

Capacity is billed on what is actively cached, not provisioned in advance. That's the model that sidesteps the FSx for Lustre pre-provisioning problem entirely. Serverless execution also runs directly alongside the filesystem, meaning agent-written code runs against the mounted data without a separate sandbox. That matters for ML teams building agentic pipelines over training data.

For teams whose workloads already span multiple clouds, or who want to stop rebuilding this layer every time a provider changes, the question stops being "S3 or EFS" and becomes "what sits in front of both."

The Storage Decisions That Will Actually Move GPU Utilization

GPUs sit idle when storage can't feed them fast enough. This is a fixable problem in most cases, and it almost never requires buying more compute.

Shard before you train. Access pattern determines S3 throughput more than any other variable. This is not optional at scale, and skipping it is the single most common way teams leave performance on the table.

Use a parallelized client. The S3 Connector for PyTorch and Mountpoint exist because naive S3 access is latency-bound at per-request granularity. A general-purpose HTTP library will not save you here.

Reserve EFS for workloads that genuinely need shared POSIX access. Not as a default because it feels familiar, and not because someone on the team used it before. The cost premium is only justified by that specific requirement.

Audit checkpoint behavior separately from training data loading. The two patterns have different optimal storage targets, and optimizing one while ignoring the other is extremely common and quietly expensive.

For inference: latency is a production concern now, not a training optimization. Storage architecture for serving needs to be designed with first-byte latency as the primary metric from the start, not bolted on after the model is already in production.

If you're running across multiple clouds or building persistent agent infrastructure over training data, the question worth sitting with is whether a cloud-agnostic POSIX layer removes the recurring cost of making this decision every time something in the stack changes. In most cases, it does.