S3 Requester Pays Buckets
What Requester Pays Buckets Are and the Billing Problem They Solve. If you share data in S3 long enough, you eventually open your …
What Requester Pays Buckets Are and the Billing Problem They Solve
If you share data in S3 long enough, you eventually open your AWS bill and think: "I don't even use this data. Why am I paying to let other people download it?" That's the moment Requester Pays clicks into place.
It's a billing configuration. Not a security feature, not a permissions system. It moves one specific line item from your bill to theirs. That's the whole thing.
Here's why that line item matters. Storing 1 TB in S3 Standard runs around $23 a month. Transferring that same terabyte out to the internet adds roughly $83. Egress costs more than triple what storage costs for frequently accessed data. Ingress, though? Free. Anyone can pour data in cheaply. Getting it out is where AWS collects.
Requester Pays flips the transfer bill. You keep paying for storage. Whoever downloads pays for the transfer and the request fees. What it is not: a new access control layer, a way to hide data, or any kind of security mechanism. Keep that mental model and the rest of this makes sense.
How the Configuration Works at the API Level
The owner side is one switch. A single bucket-level setting in the console, CLI, or API. No per-object configuration, no tagging scheme. Just flip it on.
One thing that trips people up early: if you're uploading your own data after enabling Requester Pays, use BucketOwner mode for those uploads. Otherwise your own ingestion costs show up on your bill in a confusing way. Switch to Requester mode before you publish the bucket.
The requester side is where the actual work lives. Every authenticated request has to explicitly acknowledge the charge. Not implied. Not automatic. Explicit.
HTTP header:
x-amz-request-payer: requesteron DELETE, GET, HEAD, POST, and PUT requestsREST parameter:
RequestPayerCLI flag:
--request-payer requester
Miss the header and the request gets denied. Not charged to the owner. Just rejected. It fails loudly, which is genuinely the best possible outcome when you forget something. Your pipeline breaks immediately rather than quietly billing someone the wrong thing for three weeks.
One IAM detail that matters in practice: if a requester assumes an IAM role before calling S3, the account that owns that role is what gets billed. Not necessarily the individual user. In cross-account setups, figure out whose account owns the role before you assume billing lands where you expect. It often doesn't.
Google Cloud Storage has a nearly identical mechanism where requesters supply a billing project ID with each call. Same concept, different token. If you've worked in GCS, you've already seen this pattern.
Why Anonymous Access Disappears and What That Means in Practice
The moment you enable Requester Pays, public URLs stop working. Unsigned URLs return a 403. Browser downloads break. Any script that previously ran without credentials now needs both authentication and the explicit header. Two separate requirements, and either one alone isn't enough.
The reason is straightforward: S3 needs an authenticated identity to know which account to charge. No identity means no charge attribution, so S3 just denies the request. Makes sense from a systems design standpoint. Still a surprise if you didn't plan for it.
What this means concretely:
Every requester needs an AWS account. No account, no access.
Any third party without AWS credentials is completely locked out.
Every tool, pipeline, and SDK call in your stack needs to be updated with the header.
In a large pipeline with many S3 clients scattered across services and teams, that header change is a real change management problem. Not technically hard. Just everywhere. Requester Pays is not a substitute for public open data. It's a cost-shifting arrangement between authenticated, AWS-accountable parties.
The Hard Constraint: Replication Destinations Cannot Be Requester Pays Buckets
S3 replication cannot write to a destination bucket that has Requester Pays enabled. It's a flat restriction in AWS documentation. No workaround exists inside S3 replication itself.
The constraint only applies to the destination. Your source bucket is untouched. But if you're relying on cross-region replication for durability or disaster recovery, you have to pick one: replicate to that bucket, or enable Requester Pays on it. Not both.
The reason this hurts is timing. Architects usually discover it after the bucket is already published and being used. At that point, working around it means either giving up replication or spinning up a separate bucket and rethinking the whole architecture. Neither is a fun afternoon.
If you're designing a pipeline that needs both data sharing and replication, treat this as a first-week decision, not a last-minute detail.
Where Requester Pays Is Actually Used: Open Dataset Distribution at Scale
The AWS Open Data Sponsorship Program hosts over 300 petabytes of publicly available datasets. NASA, NOAA, NIH, the EPA, and the Allen Institute for AI are all contributors. This is load-bearing infrastructure for public science, not a niche configuration.
Common Crawl is the most visible example. A single crawl snapshot covers billions of web pages and hundreds of terabytes of data, sitting in S3 and accessible to anyone with an AWS account. A large share of generative AI models released in recent years trained on Common Crawl data. If Common Crawl absorbed every egress charge from every training run, the organization would be bankrupt inside of a year. Requester Pays is what makes the whole operation viable.
Training even a modest LLM on a Common Crawl subset can mean pulling 80 to 100 terabytes of data. At standard egress rates, that's a real bill. Requester Pays puts that bill on the team doing the training, not on Common Crawl.
NASA's arrangement with AWS works the same way. High-value scientific datasets get made broadly available, and widespread research access doesn't crater NASA's budget because they're not absorbing egress for thousands of researchers downloading satellite imagery.
The pattern is consistent across all these cases: the owner has the data, wants broad access, and cannot absorb unbounded egress. Requester Pays makes distribution economically survivable for the publisher.
When Requester Pays Makes Sense for Private Data Sharing Between Organizations
The scientific dataset use case gets most of the attention. The enterprise use case is quieter but runs into the same economics.
One organization owns a large dataset. External partners need access. Without Requester Pays, every partner download is an egress charge on the owner's bill regardless of who's actually benefiting. With it, the consumer pays their own transfer costs. No invoicing, no monthly reconciliation between legal and finance teams. AWS handles the billing split.
Where it fits:
Partner data sharing across account boundaries where the owner doesn't want to subsidize access
Large data handoffs where transfer costs are significant enough to matter contractually
Situations where billing clarity between organizations matters more than convenience
Where it doesn't fit:
Internal teams inside the same AWS organization
Same-region access where transfer fees are already minimal
Any case where the data owner actually wants to subsidize access. A SaaS vendor distributing software to customers probably doesn't want their customers to need an AWS account just to download an installer.
The operational cost that gets underestimated: the requester has to instrument every single call with the header. In a large pipeline, that's a genuine change management project. Factor it in before you commit to the approach.
What Requesters Actually Pay and How the Charges Appear
The charges that land on the requester's bill are request fees (GET, PUT, LIST, and so on) and data transfer out. Storage stays on the owner's bill. Always.
S3 egress pricing in us-east-1 starts with the first 100 GB per month free, then $0.09 per GB up to about 10 TB, stepping down at higher volume thresholds. Teams pulling large datasets regularly pay meaningfully less per GB than teams pulling small amounts. Volume helps, but the meter still runs.
Cross-region reads cost more. Reading from a Requester Pays bucket in a different region than your compute adds inter-region transfer charges on top of standard egress. A Spark job reading 10 TB cross-region can add a couple hundred dollars per run in transfer fees alone, before anything else.
AI workloads make this worse because training jobs re-read data. Multiple epochs, checkpoint restores, validation passes. A dataset that gets downloaded once in a traditional workflow gets pulled five or ten times through a training loop. The meter runs for each one.
The practical takeaway is that co-locating compute in the same region as the bucket isn't optional if cost matters. It's the difference between a predictable bill and an unpleasant surprise at month end.
Requester Pays in AI Training Workflows and the Data Residency It Implies
Most major ML datasets, Common Crawl included, have primary mirrors in S3 us-east-1. That region placement isn't incidental. It quietly determines where your training infrastructure should live, whether your team realizes it or not.
The economic logic is blunt: train where the data lives. Co-locating a GPU cluster in us-east-1 eliminates cross-region transfer fees entirely. Moving the data to another region first means paying egress twice. Once to move it, then again on re-reads during training.
Checkpoints compound the problem. A single checkpoint for a large model can be hundreds of gigabytes. Across tuning runs and evaluation artifacts, AI teams routinely manage tens of terabytes of files. Each round-trip to a Requester Pays bucket adds to the bill.
HuggingFace's move to Cloudflare R2 in 2023 was a direct response to this. R2 has no egress fees, which means pulling a large dataset or model checkpoint into any cloud costs the requester nothing in transfer. That decision was a response to exactly the egress economics that Requester Pays only partially addresses.
The tension is real. Requester Pays makes large public datasets distributable for the owner. But it pushes the entire cost burden, and the infrastructure decisions that follow from it (region selection, caching strategy, storage backend), onto whoever is training. Teams that ignore this pay for it later.
The Limits of Requester Pays as a Solution to Egress Costs
Here's the thing the feature name doesn't tell you: Requester Pays doesn't reduce egress costs. It moves a line item from one AWS bill to another. The $0.09 per GB still exists. Someone just pays it.
What it doesn't do:
Doesn't reduce the actual egress fees
Doesn't change the underlying pricing structure
Doesn't help the requester avoid charges when reading at high volume
Cost attribution and cost reduction are different problems. Requester Pays solves the first one cleanly. It doesn't touch the second.
If you actually want to reduce what someone pays in egress, the options are:
Same-region compute. Eliminates cross-region transfer fees entirely.
Caching. Reduces re-reads across training epochs and repeated validation passes.
Storage backends with zero or low egress pricing. R2 and certain S3-compatible providers, for data the team controls directly.
Requester Pays is the right tool for one specific problem: public dataset distribution where the owner has no control over who accesses the data or from where. NASA, Common Crawl, NOAA. That's where it earns its place.
For everything else, reach for it only after you understand what it actually moves and what it leaves exactly where it was.
