S3 as a Message Queue Alternative

Multi-agent pipelines keep reaching for message queues, then finding the messages carry nothing but a pointer to a file that already lives in object storage. A shared filesystem built on S3 can replace the queue entirely for those handoffs. This piece maps where that swap works, where queues still win, and what the filesystem interface needs to deliver.

Why multi-agent pipelines keep reaching for message queues — and what that pattern costs

A queue solves a real problem. One agent produces a result, drops a message, and a downstream agent picks it up when ready. Neither blocks the other. Delivery guarantees, retry logic, backpressure. The queue handles all of it.

But a queue is also a broker process you have to operate, a protocol you have to integrate, and a separate state store for results once consumed. Every handoff becomes API calls, pagination, and retries that multiply as the pipeline grows. You already know the drill.

The tax specific to multi-agent systems is worse than the usual operating overhead. Intermediate results are large things (documents, structured JSON, model outputs). They don't fit comfortably inside a message payload. So teams write the results to object storage anyway, then pass only a pointer through the queue. At that point the honest question is: what is the queue doing for you?

That question frames the design problem this piece tackles. If results live in object storage either way, can a filesystem replace the queue as the coordination layer? Not by mimicking queue semantics. By making durable shared state the primitive.

How shared filesystem state works as a handoff primitive

The substitution is structural. Instead of passing a message that points to a result, an agent writes its output to a named directory. The downstream agent reads from it. The filesystem is the message. The directory convention is the routing. The storage system is the broker.

The clearest published example is the AWS Storage Blog's five-stage pipeline. Five agents (intake, risk scoring, compliance validation, report assembly, executive summarization) run on five different compute services: EC2, Lambda, EKS, ECS on Fargate, and Bedrock AgentCore Runtime. All mount the same S3 Files filesystem. Each reads from one directory and writes to the next: intake/, analyzed/, validated/, reports/, documents/, summaries/. No direct agent-to-agent communication. No queue between stages after the initial trigger.

That initial trigger is worth noting because it draws the real boundary. A document lands in S3. An event notification delivers to SQS. The EC2 intake agent picks it up. Queues handle the external boundary. The interior handoffs ride the filesystem.

Downstream stages use directory polling or scheduled triggers. EventBridge invokes both the Lambda stage and the AgentCore Runtime stage on a one-minute schedule. Long-running EKS and ECS services run a loop calling os.listdir() periodically.

The key filesystem guarantee is close-to-open consistency, which S3 Files documents: changes to a file become visible to other clients after the writer closes it and a reader opens it. You don't wait for the S3 export, which happens roughly a minute later. Downstream agents poll the mount, not the bucket.

The code reduction is concrete. Without S3 Files, each handoff calls s3.list_objects_v2() to discover files and s3.put_object() to write results. With S3 Files, agents call open() and json.dump(). The only difference between agents is the system prompt and which directories they read from and write to. What the filesystem supplies in place of queue infrastructure: durability because results survive process restarts, shared visibility because any mounted agent can read any output, directory-as-routing because conventions replace topic configuration, and deduplication through agent-side tracking of already-processed files.

Coordination without a central orchestrator: the kiro-flock pattern

Sequential pipelines are the easy case. The harder one is a fleet of concurrent agents working on independent tasks. Here S3 shared state enables self-organization without a central controller, using conditional writes as a lightweight coordination primitive.

The kiro-flock reference pattern shows the shape. Agents write append-only log entries to a shared S3 bucket describing what they did, the result, and their next intent. Other agents list and read those objects to decide their next move. No central brain. No queue manager.

The same mechanism appears in the AWS pipeline walkthrough. Before working on a document, a replica claims it by atomically creating a small marker file on the shared mount using an exclusive create. Only one replica can win that create. The others skip the document. Each document gets processed exactly once, with no message queue involved. S3 provides for free what a custom orchestrator must build: durability, virtually unlimited concurrent readers, and a flat namespace that scales without capacity planning. You don't provision capacity for "more agents reading state."

The scaling argument has teeth. As an agent fleet grows from dozens to thousands, the orchestrator bottleneck grows with it (connection limits, request queues, lock contention). Removing that central process matters more than the coordination guarantees it provides. This is a trade, not a deficiency. S3 coordination gives up strong ordering for resilience and horizontal scale.

Where S3-as-coordination breaks down and queues still win

S3-as-coordination trades strong ordering for resilience and scale. That trade is the wrong one for several common agent patterns. A central orchestrator gives strong guarantees about ordering and state because it is one process holding one view of the world. Shared S3 state explicitly gives that up. You need to be honest about when that matters.

Queues remain the right tool for strict task ordering that must be enforced, not approximated. For agents that need to negotiate in real time rather than poll for state. For workloads that cannot afford even a few seconds of staleness in what agents believe about each other. And for fan-out where a single event must reliably reach many consumers, each processing it exactly once: the pub/sub model queues handle natively. And for backpressure management: queues absorb burst traffic and let consumers process at their own rate. A filesystem has no equivalent signal for "slow down."

The debugging asymmetry deserves attention. With a central orchestrator, something goes wrong and you have one log stream and one state store. With S3 as the coordination layer, the incident timeline is scattered across however many objects agents wrote. The order you read during a postmortem may not match the order every agent wrote. You need object versioning, timestamps embedded in every state object, and a queryable event log shipped separately. None of that is free.

The use-case boundary is clear enough. S3-as-coordination fits when the agent fleet is large, tasks are independent enough that a claim-based model works, and occasional duplicate work or brief coordination lag is acceptable. It does not fit when task ordering must be strict or agents need real-time mutual awareness.

What the filesystem interface must provide to make this pattern practical

Directory-convention handoffs and conditional-write coordination only hold up if the underlying filesystem provides full POSIX semantics. Atomic rename, exclusive file creation, and close-to-open consistency are not optional features. They are load-bearing primitives.

Atomic rename gives safe handoff from a work-in-progress temp file to the visible output path. Without it, a downstream agent can pick up a partially written file. Exclusive create (O_EXCL) is how agents claim tasks without a separate lock service. Only one agent wins the create, all others skip. If agents need flock/fcntl locks on a shared file, check whether the filesystem enforces them across clients or only within one host; the AWS pipeline avoids the question by claiming work with exclusive-create markers. Close-to-open consistency means a file one agent writes and closes is visible to an agent that opens it afterward. The AWS pipeline depends on this explicitly, so check how a given layer defines and times that visibility. Concurrent mounts matter too: many agents must mount the same filesystem simultaneously. The AWS walkthrough spans five different compute environments mounting the same S3 Files filesystem at once.

Applying the pattern: when to replace the queue, when to keep it, and where to start

The right question is not "should I replace my message queue with a filesystem?" It's "which handoffs in my pipeline are passing pointers to files that already live in object storage?" Those are the handoffs where the queue is redundant overhead.

A handoff is a good candidate for filesystem replacement when the message payload is a reference to a file or object rather than a self-contained task description. When downstream agents need to read the result more than once, or multiple downstream agents need access to the same output. When the pipeline is sequential, or stages are loosely parallel, rather than requiring strict ordered delivery. And when task volume is high enough that a central orchestrator has become the bottleneck, but tasks are independent enough that claim-based coordination works.

A queue should stay when tasks must process in strict order. When fan-out to many consumers each processing the event exactly once is required. And when you need real-time agent negotiation: sub-second awareness of each other's state. Or when the team cannot absorb the operational overhead of embedded timestamps, object versioning, and a separate queryable event log.

Hybrid is the realistic steady state. Keep an event-driven trigger, SQS or equivalent, at the external boundary where a new document or task arrives. Then hand off internally through the shared filesystem. The AWS five-stage pipeline does exactly this: SQS handles the initial trigger, the first agent uses both SQS and the filesystem, and every downstream stage hands off through the filesystem.

Where to start: pick one interior handoff currently passing an S3 key through a queue. Mount the shared filesystem on both agents. Change the handoff to a directory write and a directory poll. Measure whether close-to-open consistency meets the latency requirement. The code change is small: open() and json.dump() replacing s3.put_object() and s3.list_objects_v2(). The blast radius of getting it wrong is contained to one stage.

The longer-term bet is architectural. As agent pipelines grow in depth and agent fleets grow in size, the coordination overhead of managing a broker compounds (protocol integration, broker capacity, message schema versioning). Teams that treat durable shared state as the handoff primitive from the start avoid rebuilding coordination infrastructure as they scale.