/storage_letter.

Why Shell Commands in the Cloud Lose Files and What the Storage Layer Must Guarantee Instead

Agent sandboxes need fsync durability, restart persistence, and POSIX semantics.

Sponsored by Archil

Storage Letter Editors
· 8 min read
Cover illustration for “Why Shell Commands in the Cloud Lose Files and What the Storage Layer Must Guarantee Instead”
Object Storage · November 13, 2026 · 8 min read · 1,891 words

Shell commands in cloud sandboxes lose files when storage promises more than it preserves. Serverless bash execution integrated with a persistent cloud filesystem needs three guarantees to give an AI agent a reliable working directory between runs, namely durability on fsync, persistence across restarts, and POSIX semantics. Those guarantees must hold inside the sandbox’s isolation boundary.

Why shell commands in cloud sandboxes lose files

A developer pauses an agent’s sandbox, expecting its working files to be there when it resumes. Instead, the files are gone. In a volume-only persistence model, only files explicitly saved to the mounted persistent volume survive, while memory and local files outside that volume do not. “Paused” sounded like “saved,” but the storage contract said something narrower.

Outright sandbox deletion produces a more obvious version of the same failure. The sandbox terminates, taking its local files, installed packages, and intermediate artifacts with it because none reached durable storage. The shell command may have completed correctly.

An agent downloads objects into local scratch, then loses the record of those downloads when its context window is compressed or trimmed. The files and the agent’s account of them drift apart. It may fetch the same data again or silently use a stale local copy. This is a bookkeeping failure caused by treating scratch space as a workspace whose contents must be remembered in conversation.

What "persistent" actually means across different sandbox architectures

Persistence describes what survives, for how long, and under which conditions. It is not a yes-or-no property. A persistent volume attached to an ephemeral sandbox preserves the mounted data, but the environment around it may be rebuilt. This model gives selected files on the volume a lifetime separate from the rest of the workspace, including packages and configuration outside that volume.

Snapshot-based persistence preserves a different boundary, namely a point in time. Restoring a disk snapshot returns the filesystem state captured at the snapshot’s point in time. Cloudflare’s Sandbox GA announcement described filesystem snapshots for checkpointing work and forking sessions, with rollout still forthcoming at GA. Those snapshots require an explicit trigger and do not capture memory or running processes.

A persistent sandbox backed by a persistent disk instead keeps the filesystem environment through stops and restarts. Installed packages, working files, and modified configuration can stay in the workspace when it resumes, with no need to rebuild them. A durable disk alone does not establish workspace continuity, because the sandbox must survive and remount it on resume. The word “persistent” leaves all these boundaries unstated unless the platform names them.

The three storage guarantees shell commands actually depend on

The first guarantee is durability on fsync. When that call returns successfully, the written data must already be durably committed beyond a buffer. Otherwise, the command can succeed and the file can appear present, only to disappear after a restart. This is the most misleading kind of success. A testable contract identifies the durable commit that happens before acknowledgment.

S3, GCS, Azure Blob, and R2 provide strong consistency per key, and a successful PUT means the object is durable, but they do not expose fsync. A filesystem layer must bridge that gap. If the bucket is its durable destination, the layer must complete the PUT before returning from fsync. If it copies to the bucket later, durability must already exist elsewhere, so background copying is not confused with background commitment.

The second guarantee is persistence across restarts. A stop-and-start cycle must preserve the installed packages, working directory, and modified configuration without rebuilding the environment from a base image. Durable files are not enough by themselves. An agent’s progress includes its final output and the environment it has assembled across many tool calls. Rebuilding that environment discards incremental work that long-running tasks depend on, even when a mounted data volume survives.

The third guarantee is POSIX semantics, including atomic rename, mmap, hard links, symlinks, and sparse files. Atomic rename lets a tool write a replacement and then change the filename in one indivisible operation that keeps the intermediate replacement state hidden. Rename and durability are separate promises, however. Atomic replacement does not excuse a false fsync acknowledgment, and durable bytes do not make rename atomic. Object storage provides neither these filesystem operations nor their semantics natively, so existing tools need the filesystem layer to supply them to work unmodified.

Advisory locks coordinate processes on one machine. When writers run on different machines, multi-agent workspaces need explicit ownership to coordinate them. A filesystem can behave correctly on each machine while still requiring that coordination. “POSIX” is not a distributed traffic cop.

Why isolation and storage must be designed together, not bolted together

Storage correctness depends on what surrounds the filesystem. A shared-kernel container can mount a durable volume, but the mount does not isolate its filesystem from host-level interference. A compromised container may be able to reach host storage paths. Containers offer the fastest startup among these models, but volume durability does not strengthen their kernel boundary.

gVisor puts a user-space kernel between application syscalls and the host, giving meaningful isolation without a full VM boundary. A microVM goes further by providing its own kernel, where guest filesystem semantics can be enforced before writes reach durable storage. In isolation strength, microVMs sit above gVisor, which sits above shared-kernel containers. None of those labels alone proves that fsync commits durably or that a restart preserves the environment.

Agent code with unrestricted outbound access can reach the host’s instance metadata service and retrieve IAM credentials, gaining a route around filesystem isolation. Workloads running agent code need IMDS blocked at the network policy layer. Credentials also belong outside the untrusted workload. Cloudflare’s GA implementation illustrates that separation with an outbound Workers egress proxy that adds authentication headers at the network layer while keeping tokens outside sandbox code.

What a storage layer that keeps shell commands honest looks like in practice

A concrete implementation combines a durable disk, a full POSIX filesystem, and a sandbox that resumes the same environment. The disk outlives execution independently of a last-minute save. Archil implements this pattern with a Firecracker microVM, its own kernel, and a persistent disk for each sandbox. Installed packages and working files are still there after the sandbox stops and restarts. That ties the execution environment to the storage lifetime instead of asking the agent to reconstruct the connection.

The acknowledgment point makes this architecture testable. Archil’s disk documentation states that writes are durable once fsync returns and are then copied to the backing bucket in the background. The order is the important detail here. Copying later is compatible with durability only because the write is already committed before the call returns. That differs from a FUSE layer that acknowledges the call while the only durable write is still pending.

The filesystem interface must be equally specific. For example, disks over S3, GCS, R2, or Azure Blob buckets in Archil expose atomic rename, mmap, hard links, symlinks, and sparse files so existing tools can run unmodified. Its documented locking boundary remains local to one machine, with explicit ownership needed for writers across machines. That qualification makes the compatibility claim useful.

Mounting remote data also removes the need for the agent to remember which objects it downloaded. Commands such as ls, cat, and grep can operate through the filesystem, dispensing with a manually managed scratch directory. Amazon S3 Files, launched in April 2026, takes a related approach. AWS’s announcement describes simultaneous filesystem and S3 API access to the same data without duplication. That shared access alone does not show that the sandbox meets every guarantee discussed here.

Mountpoint, gcsfuse, and blobfuse2 provide other filesystem interfaces to object storage, but the interface alone does not settle credential handling. Vercel’s documentation specifically warns that credentials passed to Mountpoint’s mount-s3 remain exposed inside the sandbox for the process’s lifetime. That warning applies specifically to Mountpoint’s FUSE implementation. For parallel attempts and rollback, disk branching is another concrete distinction. Archil documents constant-time checkpointing and branching, with a disk fork taking about 90 ms regardless of size and avoiding a full copy.

Per-tenant storage isolation for multi-tenant agent products

A shared agent platform needs the same guarantees at the tenant boundary. A tenant_id in a prompt or model-generated tool argument is an instruction to use the right data. If a tool can still reach another tenant’s files, the storage system has delegated isolation to the correctness of that instruction.

Rows filtered by tenant and objects under tenant-prefixed paths do not automatically cover the data paths introduced by agent features. Those paths include the prompt, retrieval context, tool call, and response. Each can move data from one tenant’s context to another’s when shared storage does not enforce partitioning. The storage obligation is therefore to limit what each tenant’s tools can reach.

For teams building these products, the isolation unit should be a tenant’s disk or separately scoped bucket mount. Session data should also be partitioned by tenant in object storage. Data needs at least a tenant-and-session prefix to keep it organized, while a separate bucket for each tenant gives it a stronger boundary and removes prefix-traversal risk. The distinction is between naming a partition and enforcing one. A misconfigured tool call should not be able to turn one tenant’s path into another tenant’s access.

That design also needs workable storage economics. Archil’s per-disk billing on active data supports separate disks at thousands of tenants without a flat storage charge for every tenant. A tenant whose agent writes nothing incurs no active-data storage usage. That does not mean the platform has no plan minimums. Unused tenant disks need not come with a separate fixed storage bill that pushes teams to share storage with a weaker boundary.

Evaluating a sandbox's storage guarantees before you commit to it

Ask the vendor whether fsync returns only after durable commitment, and ask it to name the commitment behind its claim of “durable storage”. Then establish whether packages, configuration, and the working directory survive a stop-and-start cycle. A surviving volume is not sufficient evidence. The vendor must say whether the environment resumes or is rebuilt around the data that was kept.

Next, ask about atomic rename, mmap, fsync, hard links, symlinks, and sparse files, including which tools have been tested unmodified. Ask where advisory locking stops and how writers on different machines get explicit ownership. For a multi-tenant product, establish whether disk or bucket boundaries prevent one tenant’s tool from accessing another tenant’s data. A prefix convention alone does not answer that question.

Is outbound access disabled until allowed, or enabled by default, and is the instance metadata service blocked? Are secrets injected at the network boundary, as in the egress-proxy pattern, or handed to sandbox code? Storage permissions can be bypassed through exposed credentials. A durable, POSIX-compatible disk is still an incomplete answer if the workload can obtain access beyond it.

Finally, ask how long disk forks or snapshots take and whether that time grows with disk size. Establish whether an existing S3, GCS, R2, or Azure Blob bucket can be mounted as a POSIX filesystem without copying its data. Those answers show whether parallel attempts will work in practice and whether the agent will have to keep track of scratch downloads again. Specific answers are the standard. A storage layer that cannot provide them is making the same implicit promises that turned “paused” into lost work.

This article is sponsored by Archil.

Sources

  1. AI Code Sandboxes: A Comparative Security Study. Part 1 of 2 -- Engine-Level Properties (Attack Surface, Leakage, Stackability, CVE History, Patch Cadence, Fuzzing)
Filed underObject Storage

More in Object Storage