Approaches to Eliminating Cold-Start Latency in S3-Intensive Pipelines

Engineers will spend three weeks shaving 40 milliseconds off a model's inference loop, then shrug when the instance sits idle for two full minutes before that loop even starts. That's the joke hiding inside most "cold start" complaints: the slow part usually isn't the model. Cold-start latency in S3-intensive pipelines is a stack of independent bottlenecks, each one living at a different layer of the infrastructure, and treating them as a single problem is why so many fixes don't hold up in production. Three separate delay sources get bundled under one complaint. First-touch block fetches happen when an instance boots from an EBS Snapshot: AWS creates a sparse volume and pulls data from S3 block-by-block only as it's read, so a pipeline loading a large model into GPU VRAM hits severely throttled I/O instead of the full disk speed it was promised. Sequential retrieval chains in agentic and RAG workloads stack up hundreds of milliseconds to over a second of delay before a single token appears, because a traditional cold path (fetch from S3, chunk and embed, run vector search, then prefill the LLM) sequences these steps one after another. Network-bound weight ingestion hits at inference time, when tens to hundreds of gigabytes of model weights have to move from remote object storage over a standard network interface, throttling memory bandwidth so badly that load time dominates total cost even once the compute is sitting there ready to go.

The physics behind all three is the same, and it's unforgiving. Cloud object storage runs at 50 to 200 milliseconds of latency for random access, while local NVMe ephemeral storage runs at 5 to 50 microseconds, a gap of roughly three orders of magnitude. That gap leaves GPU compute idling while it waits on storage, which turns a performance quirk into a line item on the cloud bill that often goes unnoticed until someone profiles the pipeline end to end. A GPU cluster burning rental hours while waiting for bytes to arrive is like paying for a chef, a kitchen, and a dining room while the delivery truck is still stuck in traffic.

EBS lazy loading and its silent throughput cost on GPU instances

EBS lazy loading is the sneakiest of the three, mostly because teams think they've already solved it. Baking a model into a custom AMI feels like finishing the job; it's actually the setup for the next surprise. When AWS launches an instance from an AMI backed by an EBS Snapshot, it doesn't copy the data onto physical disk right away. It creates a sparse volume, and the real data gets pulled from S3 to disk block-by-block, one read at a time, the first time the application asks for it. This is EBS Lazy Loading, also known as First Touch Latency.

An application tries to load a large model suite into GPU VRAM, I/O spikes as expected, but because the data isn't physically sitting on the disk yet, read speed gets throttled far below what the volume is rated for. The instance looks awake. It performs like it just rolled out of bed.

A real version of this played out on an asynchronous image processing pipeline running on AWS Batch, using g6.xlarge and g6.4xlarge GPU instances, a Triton Inference Server Docker image around 30 GB, and roughly 150 GB of AI models. The team baked everything into a custom AMI, assuming that would eliminate the delay. Cold-start lag stuck around anyway, because the EBS volume was still sparse the moment the instance launched. Baking the image in advance just moves the problem of pre-populating the disk somewhere less visible.

The obvious next move, downloading everything at boot, doesn't fix it either. Pulling tens of gigabytes from S3 during startup runs headfirst into EBS write throughput limits, and that leaves a GPU instance (the expensive kind, billed by the hour whether it's working or not) sitting idle for many minutes before it processes a single file. So if baking the model into the AMI doesn't solve it, and downloading it at boot doesn't solve it, the volume itself may be the wrong place to put the model at all.

Using S3 Mountpoint streaming to bypass EBS entirely

The fix is skipping the volume for model weights entirely, by mounting S3 directly as a file system the instance reads from the moment it boots. Instead of asking "how do we get the data onto disk faster," the better question is "why does it need to be on disk at all." A handful of tools are built around exactly that filesystem-first logic, treating the filesystem layer itself as the primary interface rather than a cache sitting in front of object storage.

AWS's own version of this is Mountpoint for Amazon S3, a high-performance open-source file client that mounts an S3 bucket as a local file system, tuned for high-throughput read workloads like ML training and inference. Startup time effectively becomes instant, because the directory is visible and usable on boot without any download step at all. The operational upside stacks up cleanly: it bypasses EBS disk throughput limits entirely. It also uses the EC2 instance's full network bandwidth, up to 12.5 Gbps on a g6.xlarge, instead of being capped by how fast EBS can write. And it streams model data directly from S3 into memory on demand, so only the data the application actually touches is transferred.

AWS pushed this pattern further with S3 Files, which went live in April 2026. S3 Files lets S3 buckets act as POSIX-compliant file systems over NFS, backed by a fully managed elastic file system layer. On first access, it builds out metadata for objects in that namespace so directory traversal stays fast; small files can get loaded into the file layer during that access, while larger objects stream directly from S3 without ever being fully materialized on disk. The underlying idea connects both approaches: applications don't need to see all their data before they start running. They need the interface to behave as though the data is already local, while the real transfer happens quietly in the background. That's the same logic that makes a POSIX filesystem interface valuable for AI workloads generally, since existing code runs unmodified, with no SDK rewrite required.

The three-tier caching architecture as the production standard for training pipelines

Single-instance boot problems and multi-GPU training problems aren't the same animal, even though they both get called "cold starts." For an active training run spread across a cluster, the fix is moving data progressively closer to compute before the GPU ever asks for it, and the three-tier caching strategy is the architecture that pulls that off without stalling the pipeline. The pattern runs S3 or GCS for durable checkpoints, shared network storage or NVMe for the datasets in active use during training, and local NVMe for whatever batch is currently being processed. As of April 2026, this remains the production standard across PyTorch FSDP, DeepSpeed, and Ray Train.

Prefetching, not the tiering by itself, is what actually does the work. Before the GPU finishes chewing through the current batch, the data loader spins up a background thread that fetches the next several batches from S3 into local NVMe using asynchronous I/O, so the data is already sitting there by the time the GPU asks for it. In practice, that means reading the next two or three batches while the current one is still processing, which hides the latency almost completely. It means keeping shuffle buffers on NVMe rather than regenerating random access patterns over the network every time the data gets reshuffled. And at larger scale, it means running distributed data pipelines with parallel interleave across Kubernetes DaemonSets, with each node responsible for prefetching its own shard.

It's worth being honest that none of this is free, and here is where it stops working. Clusters past a certain size run into the I/O wall regardless of how good the prefetching logic is: S3 list operations start timing out, and throttling kicks in once thousands of parallel requests hit the bucket at once. The standard workaround is to stop looping on ListObjectsV2 and use S3 Inventory or a DynamoDB-backed file manifest instead. Checkpointing is its own quiet tax on this system: saving a large model's state to network NFS on a fixed schedule can eat a real chunk of wall-clock time per epoch, and the standard mitigation is async checkpointing run on a separate thread pool so it doesn't block training. And caching, however well-tuned, can't rescue a dataset that's scattered across tens of thousands of small S3 files. The metadata overhead alone is enough to sink throughput before the first batch even loads. Consolidating that data into properly sized shards, typically 100 MB to 1 GB, with sequential access patterns, isn't an optional tuning step here.

GPUDirect Storage and NVMe locality in weight loading

When a model's weights need to land on a GPU, the binding constraint is the CPU bounce buffer in the I/O path, not the network or the model's precision format. Quantization still matters, for memory footprint and for inference efficiency once the model is loaded. It just isn't the lever that controls how long loading takes.

GPUDirect Storage (GDS) solves the actual bottleneck by enabling direct NVMe-to-GPU DMA, skipping CPU memory out of the path entirely. The payoff is concrete: a sharded GDS setup loads a large frontier model's FP8 weights in an 8-way tensor parallel configuration across 8 GPUs in 6.4 seconds. The comparison that makes the point land is almost unfair to NFS: over NFS at limited bandwidth, a small INT4 model, already compressed down for speed, takes many seconds to load. The same model, sitting on local PCIe Gen4 NVMe, loads in roughly a second. The bottleneck is the I/O path, not the data volume. Quantizing a model is like packing lighter for a flight that's stuck on the tarmac anyway.

Infrastructure providers have built around this same insight in different ways. Google Cloud's Managed Lustre now delivers bandwidth at a scale dramatically improved over the prior year, well ahead of other hyperscalers, with very large capacity on offer; paired with the run:AI Model Streamer and Rapid Cache, models load significantly faster, removing a storage bottleneck that used to be taken for granted. CoreWeave takes a different route for AI workloads, combining an InfiniBand backbone with Local Object Transport Accelerator (LOTA) caching for high per-GPU throughput, while running a unified data plane across regions so replication lag and manual synchronization stop being sources of cold-start delay in the first place. Different hardware, same underlying bet: locality beats cleverness.

Proactive cache warming and ingest-on-write to eliminate the first-read miss

Every caching system built so far in this piece shares one weak spot: the first read is always cold. No matter how fast the prefetching or how smart the tiering, the very first request for a given piece of data has to go the slow way, because nothing's been cached yet. The fix for that weak spot isn't a faster reaction to the miss, it's removing the possibility of the miss by making sure that anything written to object storage gets written into the cache at the same moment, so the first read is a hit instead of a gamble.

Google Cloud Storage's Rapid tier does exactly this with a feature called "ingest on write": data gets written to Rapid Cache at the same time it's written to the Cloud Storage bucket, so the very first read gets an immediate cache hit, and checkpoint restores come back meaningfully faster than they would from traditional object storage. Google extended the idea with Rapid Bucket, announced at Next '26, which carries the same approach further, cutting blocked GPU time and speeding up data loading for multi-modal training runs, checkpoint restores, and writes, again compared to traditional object storage.

The principle here travels well beyond any one cloud provider. Any architecture where the write path and the cache-fill path are the same operation eliminates an entire category of cold start, the kind caused by data that's technically present in the bucket but absent from the fast tier the GPU actually reads from. That's the structural argument for treating cache not as a separate layer that needs to be warmed up after the fact, but as a first-class participant in every write that happens. A restaurant that preps ingredients the moment the delivery truck pulls up, before anyone's even hungry, gets ahead of orders in a way that prepping as orders come in never can.

Co-location's trade-off between solving cold starts and creating portability debt

Every fix in this piece, from Mountpoint streaming to GPUDirect Storage to ingest-on-write caching, shares a quiet assumption: that compute and storage sit close enough, or are tied tightly enough, to make the fast path possible at all. That closeness is the whole point and also the catch. The tighter that binding gets, the harder it becomes to pick the workload up and run it somewhere else without rebuilding a meaningful chunk of the pipeline.

That's not an argument against co-location. Pulling model weights to a GPU in 6.4 seconds instead of two minutes is worth something real, and so is a training cluster that never stalls on a checkpoint write. But each of the fixes described here, custom AMIs, Mountpoint mounts, three-tier NVMe caching, GPUDirect Storage, ingest-on-write buckets, comes with a degree of attachment to the provider or stack that implements it. A team optimizing hard for cold-start speed on one cloud's storage and compute pairing is, often without noticing it happen, signing up for a harder migration later if business needs change. The engineering tradeoff is plain enough to state directly: speed now, bought with flexibility later. Matching the right fix to the right layer, as this piece has tried to lay out, works best when that tradeoff gets named up front rather than discovered the day someone tries to move the workload somewhere else.