Iterative Context Discovery vs Prompt Front-Loading

Two ways exist to hand an agent a task. Either you pack the context window full before it starts working, or you let the agent go find what it needs while it works. That's the whole decision, and it's not a small one, because it shapes retrieval design, token spend, latency, and how storage gets built underneath all of it.

Prompt front-loading is the instinct most teams reach for first: gather the documents, the memories, the tool definitions, the instructions, cram it all into the prompt, and hit go. Iterative context discovery flips that. The agent starts light, almost bare, and pulls in information turn by turn as the task actually demands it. Anthropic has called this the natural next step in prompt engineering, and that framing is right: it's a shift in what question you're even asking. The question moves from "what do I say to the model" toward "what fills its window, and when does that happen."

Front-loading feels safer, since you control what goes in, you can inspect it, and you know exactly what the model saw before it opened its mouth. Discovery feels riskier and looser, harder to pin down. Here's the thing though: the safer-feeling option is the one that breaks down at scale, and the rest of this piece is about why.

What prompt front-loading actually does to a model's reasoning

The logic behind front-loading sounds reasonable on paper: give the model everything it might need, don't make it ask twice. Turns out that logic doesn't survive contact with how these models actually process what's in front of them.

One study sharded benchmark prompts across multiple conversation turns, mimicking the way context actually gets assembled in real use, rather than dropping it all in one clean block. Average performance dropped sharply across every model tested, including frontier reasoning models. The mechanism is almost embarrassing once you see it, because early, incomplete answers the model gives stick around in context and poison the reasoning steps that come after. The context contaminates itself.

There's a name for the broader pattern: context rot. Output quality degrades as context length grows, and this happens whether or not you're anywhere near the window limit. The pattern holds broadly: output quality degrades as context length grows, and this happens whether or not you're anywhere near the window limit.

The strangest part of that finding is that models did better on shuffled text than on coherent text. Coherent, well-structured input creates positional patterns, and those patterns nudge the model toward recency bias, meaning it pays more attention to what's near the edges of the window and less to what's buried in the middle. Loading up the middle of your context with important material puts your best information where the model is least likely to look.

None of this is a rare edge case you can engineer around, since it's baked into how attention works. Adding more context to cover your bases can actively work against the reasoning you're trying to support.

The token economics that make front-loading expensive at production scale

Long context windows tempt teams into a bad habit: if the model can technically hold a million tokens, why not just hand it everything and let it sort things out? Sorting things out costs money, and that's why.

RAG systems fall into this trap constantly. Even a modest set of retrieved documents is already a wall of text before the model even gets to the actual question being asked. System prompts make it worse in production agents: behavior rules, constraints, tool definitions, formatting instructions, all of it repeats on every single API call across a multi-turn chat. That overhead compounds with every exchange.

A 2025 arXiv study looked at 500 real conversations averaging 101,601 tokens of context and put a number on the gap. Memory-based retrieval, pulling only the top 20 relevant chunks (about 1,046 tokens per query), cost $0.65 total across 500 evaluation questions, or roughly $0.0013 per query. Running the same model against the full conversation history instead cost $14.79 for 504 requests, or $0.0293 per request. Same task, same model, and roughly 22 times the cost, with the only variable that changed being how the context got assembled.

Model pricing spreads make the stakes worse. As of mid-2026, 13 models ship with windows over 1 million tokens. Fill that window on one model and it costs $0.14, while filling the same window on another runs $10.00. That's a 71x spread, and it stacks directly on top of whatever inefficiency your architecture is already carrying. Underneath all of this sits a hard mathematical wall: attention cost grows quadratically with sequence length, so doubling your context roughly quadruples the compute and memory needed to process it. At training scale, that's not a rounding error, but a wall you hit.

How iterative context discovery works as a loop, not a setup step

Picture the agent starting with almost nothing: a task description, maybe some stored user memories, a small set of tools. Each turn, it looks at what the current query actually needs, and the framework assembles context for that specific call.

Tool output and model output get appended as the turn happens. Anything new the agent learns along the way gets written to storage that persists. The loop keeps going until some terminal condition hits, and the context grows because the work demanded it.

The trick underneath this is keeping lightweight references around instead of full objects: a file path instead of the file's contents, a stored query instead of its results, a document ID instead of the document. The agent loads the real data at runtime, through a tool call, only when it actually needs it.

Claude Code runs on exactly this pattern. It writes targeted queries, stores intermediate results, and uses plain bash commands like head and tail to peek at large files without pulling the whole thing into context. What ends up in the window reflects what the task actually surfaced.

There's academic backing here too: framing iterative retrieval as a decision process, trained through policy optimization against feedback from the model itself, has shown real gains over baseline approaches on semantic parsing tasks. There's also a natural ceiling to how much iteration helps, since accuracy climbs steadily through a few rounds of refinement, then flattens out around the fourth iteration, with near-best performance often reachable using well under 5% of the available prompt budget. The system tells you when to stop.

The tool design rule that falls out of all this: each tool should do one thing, return only what that step needs, and leave no ambiguity about its scope. Build the smallest surface that works, not the biggest one you can imagine needing.

Why the tool surface an agent carries is itself a context problem

Every tool definition sitting in an agent's prompt eats tokens. That's space the actual task doesn't get to use, and it's one more thing the model has to reason through before it can act on anything.

Tool sprawl is context bloat wearing a disguise. Add a bespoke tool for every data source, every API, every specific operation, and you've added prompt overhead that repeats on every single call, forever, whether that tool gets used or not.

There's a cleaner way to do it: lean on interfaces the model already knows cold. Bash is one of those. Frontier models have seen enormous volumes of command-line interaction during training, so they arrive already fluent: ls, cat, head, grep, find, file paths. None of that needs to be taught through a custom tool schema, because the model already speaks that language. One well-understood interface can replace a sprawling shelf of purpose-built tools.

This isn't purely about saving tokens either, since it's also about how much the model has to think before it acts. Fewer tools with sharper, clearer meaning means less time spent guessing which one applies. The goal is a small, stable set that stays clear about what each piece does.

What iterative discovery requires from the storage layer

Front-loading treats storage like a tank you drain before the real work starts: retrieve, load, go. Iterative discovery treats it like a room the agent keeps walking back into, over and over, for the entire length of the task. That's a fundamentally different demand on the storage layer, not just a different retrieval strategy on top of the same one.

A few things fall out of that directly. The agent needs fast, random access to specific files or byte ranges, since full sequential reads aren't the common case anymore. It needs real POSIX behavior, meaning atomic rename, proper locking, fsync, so that writing an intermediate result and reading it back doesn't corrupt when two runs overlap or a session gets cut off mid-task. It needs a shared workspace that survives across sessions; if every new session starts from a blank slate, the whole advantage of discovery evaporates. And it needs room to grow without being told in advance how much room it'll need, because nobody knows at the start whether a task needs 1 megabyte or 1 gigabyte of working data.

Cloud object storage (S3, GCS, and the like) is where most of this data already lives, and it's not going anywhere. It just wasn't built for the latency profile that iterative access demands, so closing that gap without forcing a full copy or migration is the actual infrastructure problem this creates. Mounting an existing bucket as a POSIX volume, backed by NVMe caching for fast reads while the bucket itself stays the source of truth, is one way to close it: agents get to work through familiar file operations without anyone standing up a separate pipeline just to move data into reach.

The training infrastructure backdrop: why data movement costs are already a known bottleneck

This isn't a new problem, since training infrastructure hit it first.

Per Epoch AI, training compute for frontier models has grown roughly 4 to 5 times a year between 2010 and 2024. The infrastructure carrying that compute hasn't kept pace, and the gap shows up as wasted time, not wasted flops. The GR00T-N1.5 benchmark makes this concrete: restructuring the data pipeline for a training run across a thousand GPUs cut a single training round from 15 hours down to 22 minutes. A 40-fold speedup came entirely from fixing how data moved.

Only about a quarter of AI initiatives deliver on their ROI expectations, and the model is rarely the reason. The infrastructure sitting around it, and specifically, data systems that are disconnected and fragmented across teams and providers, carries more of the blame. The public cloud still holds most AI training data (roughly 68%), but compute itself has scattered. GPU-as-a-service now makes up about 40% of workloads, which means multi-provider setups are common, and that fragments exactly where data needs to be reachable from.

The lesson carries straight over to inference and agent work. Teams that stop hand-building custom pipelines to shuttle data around, and instead make storage reachable directly from wherever compute happens to sit, get back real time and real money.

How agent infrastructure should be designed if context is discovered, not delivered

If context gets assembled on the fly, storage sits directly in the loop. Every bit of latency there is latency the agent feels on every single turn, not just once at startup.

A few principles follow from that. Storage access should work the same no matter which cloud it's on: S3, GCS, R2, Azure Blob, mounted the same way, so an agent moving between providers doesn't drag a data migration along with it. Compute should sit next to the filesystem; running bash commands or agent-written code should happen right where the data already lives, instead of pulling data across to some separate sandbox first. A serverless environment that runs bash directly against a mounted filesystem collapses the tool surface and removes the sandbox as yet another system somebody has to babysit.

Context needs to persist too, across sessions and across parallel runs, in a shared workspace that doesn't reset every time a conversation ends. The bucket underneath all of it should stay the source of truth, never a copy quietly held somewhere by a vendor; agents write into their workspace, the object storage stays authoritative, and access can get revoked without losing anything or migrating anything. And because working set sizes are unpredictable by nature, billing should track actual cache use rather than some capacity provisioned in advance, since that's the pattern agentic work actually follows.

None of this is harder than front-loading, but it is a different bet, and the teams already ahead are the ones who quit treating storage like a loading dock and started treating it like the agent's desk.