Ingress vs Egress Costs in S3 Data Pipeline Design

AWS has a dirty little secret baked into its pricing model, and it's not subtle once you see it. Moving data into S3 is free. Moving it out costs between $0.09 and $0.19 per gigabyte, depending on where it's going. That asymmetry, right there, is the single most underestimated design constraint in cloud data pipeline work.

The industry has a name for this: data gravity. Once your data lands in a cloud, the cost to pull it back out creates structural stickiness. It's not an accident. Egress pricing functions as a lock-in mechanism, and according to research from Akave, a majority of IT leaders cite egress costs as the single biggest barrier to switching providers. The pricing is doing exactly what it's designed to do.

Azure makes the math painfully visible: it costs roughly $0.018 per GB per month to store data, and $0.087 per GB to download it. That's nearly a 5x gap between storing a byte and moving it. AWS follows a similar logic. The $0.09/GB rate for the first 10TB of monthly internet egress is the number most engineers encounter first, and it's the baseline this piece works from.

Here's the frame that matters: this is not a complaint about pricing. Egress asymmetry is the fixed parameter. Pipeline architecture is the variable. The goal is to treat egress exposure the same way you'd treat latency or fault tolerance. Design around it deliberately, or pay for it accidentally.

The Paths Through AWS Where Egress Is Charged, Free, or Discounted

Not all data movement costs the same. The difference between zero dollars and $0.09/GB comes down to one thing: where your compute sits relative to your bucket.

Free paths:

• S3 to EC2, Lambda, ECS, EKS, RDS, and most other AWS services in the same region: $0.00
• S3 to CloudFront, within the same account and region: $0.00 (AWS quietly eliminated a prior charge here in late 2024. Worth knowing.)

Charged paths:

• Internet egress: $0.09/GB for the first 10TB/month
• Cross-region transfer, most US corridors: $0.02/GB
• Cross-AZ: $0.01/GB each way. Small per hop, invisible until it isn't
• NAT Gateway: this one bites people constantly

The NAT Gateway trap deserves its own callout. If your EC2 instance sits in a private subnet and reads S3 through the default route, you pay $0.045/GB for data processed through the gateway, plus $0.045/GB for the transfer itself. That's $0.09/GB total. The same rate as full internet egress, for traffic that never left AWS.

The fix is a VPC Gateway Endpoint for S3. It's free to create. The cost is architectural inattention, not architectural complexity.

The practical implication here is bigger than it looks. Compute region and subnet routing aren't just performance decisions. They're egress decisions. Every choice about where a function runs or how a network is configured has a price tag attached.

What Egress Actually Costs When a Pipeline Runs at Scale

Let's do the math that most teams skip until the bill arrives.

A team moving 15TB per month out of S3 is paying $1,325 at current rates. That's typically more than the monthly storage cost for the same data. Egress accounts for somewhere between 6 and 12 percent of typical cloud bills, per CloudZero's 2025 analysis. But that average hides the shape of data-heavy workloads, where the ratio flips entirely.

Here are two patterns that show up constantly:

A 50TB analytics dataset with 20TB of monthly reads. Storage for 50TB runs roughly $1,150/month. Egress on 20TB runs roughly 60% of the total bill. The "storage" problem is actually a movement problem.

A SaaS product serving 10TB/month to the internet. Egress alone hits $912.60. Standard storage for that dataset is $235.52. Egress is nearly 4x the storage line.

The pattern is consistent: cheap cloud storage becomes expensive the moment workloads actively consume it. Storage-cost comparisons between S3 and alternatives like Backblaze B2 or Wasabi look dramatic at face value. Once egress enters the picture, total monthly spend on S3 can run 10 to 17 times higher. The delta is almost entirely data movement, not storage.

The implication for anyone designing a pipeline: estimate the egress line before architecture is locked. Not after the first bill. Before.

How AI Training Turns Egress from a Line Item into a Compounding Multiplier

Standard pipelines read data roughly once. AI training reads the same data repeatedly across epochs, checkpoints, and distributed workers. That changes everything about how egress compounds.

The rule of thumb: total data moved during training is often 10 to 100 times the raw dataset size.

Per Akave's analysis, a typical enterprise AI project uses somewhere between 10GB and 10TB of training data. With the read multiplier applied, actual data movement lands between 30GB and 30TB. At $0.09/GB, egress for a single model can reach $2,700. Akave also found that 62% of organizations exceeded their cloud storage budgets in 2024. The primary driver was data movement, not storage pricing.

Inference adds a separate pattern worth thinking through. A 70B FP16 model is roughly 140GB. If weights are pulled from S3 on each container restart, which is common in ECS and EKS deployments without persistent volumes, that's $378 per month in egress at one restart per day. For a single model. Running at modest scale.

The compute cost framing puts this in context. GPU hours account for 70 to 85 percent of AI training costs. Storage, data transfer, and tooling add another 15 to 30 percent on top. A $50 apparent GPU job can cost $65 to $80 fully loaded. The egress line doesn't look like the problem until you add it up.

AI training is the workload where the read multiplier makes the egress line outpace every other infrastructure assumption made at design time. It's a corner case for standard pipelines but the default behavior of this workload.

Pipeline Design Decisions That Control Egress Exposure Before Writing a Line of Code

The most durable egress reduction is positional. Compute co-located with storage in the same AWS region pays $0.00 on S3 reads. A function one region away pays $0.02/GB on every single call. That rate doesn't sound like much. It adds up fast at pipeline scale.

Within-region routing discipline:

• Use VPC Gateway Endpoints for S3. Eliminates the NAT Gateway path and its hidden $0.09/GB effective charge.
• Keep pipeline stages in the same availability zone where feasible. Cross-AZ transfer adds $0.01/GB each direction, and it's completely invisible until billing.

Data format selection as an egress decision:

This one is underappreciated. Columnar formats like Parquet enable column pruning and predicate pushdown. Only the columns and rows a query actually needs cross the network. That's a direct reduction in billable bytes, and a performance win.

S3 Select takes this further by pushing filters into S3 itself. The object doesn't leave the bucket in full. Only the matching subset does.

Compression on text content (Brotli or gzip) reduces object size by 60 to 80 percent before storage. Since egress is charged per byte, compression savings flow directly to the transfer line. This is one of the highest-leverage, lowest-effort interventions available.

Output design:

Every intermediate result written to S3 and read by the next pipeline stage is a potential egress event. Minimizing stage handoffs, or keeping them within-region, shapes the bill more than most single-point optimizations.

CloudFront for user-facing reads:

Since the S3-to-CloudFront transfer charge was removed in late 2024, CDN offload is the most straightforward way to serve frequent reads without per-GB charges. CDNs, same-region deployment, and compression can collectively cut egress costs by 40 to 60 percent.

Caching and Local NVMe as the Architectural Answer to Repeated Reads

The core insight is simple. Egress compounds because the same bytes get fetched repeatedly. A cache collapses N fetches into one. You pay egress once.

Meta found that a majority of GPU cycles stalled waiting on training data, even with exabytes stored in their own distributed filesystem. Storage capacity local to training hardware was the bottleneck, not network speed. If that's true at Meta's scale with their own infrastructure, it's definitely true for teams reading from S3 across a network boundary.

Local NVMe caching is particularly effective for multi-epoch training. Cache the dataset on first read. Pay egress once. Serve subsequent epochs from local storage. AWS Mountpoint for S3 supports this pattern natively, offering built-in caching to instance storage (NVMe), EBS, or memory. The S3 bucket stays as the source of truth. Repeated reads serve locally.

S3 Express One Zone for AI workloads:

It costs noticeably more per GB per month to store data here, which is roughly several times Standard pricing. After AWS's April 2025 price cuts, GET and PUT costs dropped substantially. The payoff is consistent single-digit millisecond latency. SageMaker training jobs using co-located Express One Zone storage have shown 40 to 60% shorter training times from eliminated I/O wait.

If GPU idle time is your binding constraint, the storage premium is small against the compute cost saved. That math usually works out.

RDMA for S3-compatible storage:

NVIDIA's RDMA libraries allow AI workloads on GPU nodes to access object storage faster than standard TCP, improving GPU utilization without physically moving data. Cloudian, Dell, and HPE have adopted this in their high-performance storage products.

The design principle is this: egress is a per-fetch tax. Caching is a one-time payment that amortizes across every subsequent read. At AI training scale, that amortization is the difference between a reasonable bill and an embarrassing one.

Observability Gaps That Let Egress Costs Accumulate Unnoticed

Here's the structural problem. Egress charges appear on a consolidated bill, rather than next to the pipeline stage that generated them. There's no native per-bucket or per-request egress line item by default. Most teams discover egress costs after they've spent the money, not before.

What actually helps:

• S3 Storage Lens gives per-bucket and per-prefix activity metrics. As of December 2025, those metrics can be exported directly to S3 Tables with Iceberg support and queried through Athena. That means SQL analysis of storage and transfer trends over time, which is a meaningful step up from staring at a billing chart.
• Cost Allocation Tags on buckets and prefixes can surface which pipeline stages or teams are driving egress. This requires intentional tagging before costs accumulate. It doesn't work retroactively.
• CloudWatch and AWS Cost Explorer show regional transfer totals but don't attribute them to specific S3 operations without additional instrumentation (S3 server access logs or CloudTrail data events).

The NAT Gateway trap is the canonical observability failure. That $0.045/GB charge appears under "Data Transfer" in EC2 billing, not S3. Engineers auditing S3 costs never see it. It's sitting in a different line of the bill, invisible unless you know to look.

The practical minimum before any pipeline goes to production: enable S3 Storage Lens at account level and set up a Cost Explorer filter for "Data Transfer." Catching routing errors early, meaning NAT Gateway paths or compute running in the wrong region, is dramatically cheaper than discovering them after six months of billing.

How to Evaluate a Pipeline Architecture Against Its Egress Exposure Before It Ships

The egress audit starts with the data flow, not the cost. Map where data enters, where it's read and by what service, and where it exits the AWS boundary. Every arrow on that diagram has a price from the rate card. The architecture review should price each arrow before anything gets built.

Four questions for every pipeline stage:

1. Is the compute reading this data in the same region as the bucket? If not, add $0.02/GB minimum on every read.
2. Is S3 accessed through a VPC Gateway Endpoint, or through NAT? NAT adds an effective $0.09/GB in charges that won't appear on the S3 line.
3. Does this stage read the same data more than once, across epochs, restarts, or parallel workers? If yes, caching or Express One Zone should be part of the conversation.
4. Does this stage's output leave the AWS boundary, to users, external systems, or another cloud? If yes, this is the most expensive hop in the pipeline, full stop.

On AI workloads specifically: use the 10 to 100x read multiplier, not the raw dataset size, as the basis for the egress projection. Estimating against the dataset size alone is almost always wrong.

On format and compression: these decisions belong in the design review, not the optimization backlog. Parquet column pruning, S3 Select, and compression are far cheaper to adopt at schema design time than to retrofit after data is already in production.

One last thing worth saying directly. The asymmetry between ingress and egress is not going away. Providers price egress to recover network infrastructure costs and to maintain switching friction. Both motivations are durable. Designing around egress exposure is a permanent skill, not a workaround for a temporary pricing quirk. The engineers who internalize it early stop being surprised by bills. And that's genuinely worth something.