Solutions for Eliminating Cold-Start Latency in S3-Backed Compute Fleets

Cold-start latency in an S3-backed fleet is a sequence of separate bottlenecks stacked one after another, each of which has to be diagnosed and fixed on its own terms. A model server has to clear five distinct steps before it can answer a single request: pull the container image, load the model weights into GPU memory, warm up the CUDA kernels, compile the CUDA graphs, and register itself with service discovery [3][1]. E|S3 sits underneath most of these as the canonical source of truth, and that dependency turns it into a recurring drag rather than a one-time cost.

The June 2026 paper "Breaking the Ice: Analyzing Cold Start Latency in vLLM," by Kabakibo, Trivedi, and Wang, gives the first systematic breakdown of what happens during vLLM startup. It splits the process into six foundational steps and finds the whole thing runs predominantly CPU-bound. The bottleneck moves depending on which stage you're in, rather than sitting still in one place waiting to be optimized away. The multi-stage structure explains a pattern the paper documents: startup latency varies by more than 4× across major vLLM releases, with a 2× reduction between v0.9 and v0.10, because changes to individual components propagate unevenly through the chain.

That's the structural reason a single fix never sticks. Serverless deployments make this brutally visible: scale infrastructure to zero, and the very first request after any idle period has to pay for the entire chain, start to finish. A cold-start GPU instance can take more than 40 seconds to produce its first token, while a warm instance is already generating tokens roughly every 30 milliseconds. Per-token throughput was never the problem. The chain in front of it is.

P|The rising cost of tolerating this chain

At small scale, a slow cold start is an annoyance a user might not even notice. At fleet scale, it is a line item. Once a chain like this compounds across thousands of servers, its aggregate cost stops looking like a rounding error: it consumes a measurable fraction of total GPU time that represents real money and real SLA risk.

The BootSeer study (arxiv 2507.12619) tracked more than 28,000 training jobs across tens of thousands of GPUs over one week and found that startup overhead alone burns a small but significant share of total GPU time in one production training cluster. And that waste doesn't just track with how big a given job is. It scales with the size of the fleet itself, so the bigger the operation gets, the worse the tax gets. R|GPU training is I/O-bound work wearing a compute-bound reputation. J|The storage layer decides whether an expensive GPU cluster sits there burning electricity without producing anything.

Inference-serving fleets feel a sharper version of the same pain. During a cold start, GPUs are provisioned and billed but sit idle, generating no tokens and serving nobody. That idle window is exactly when SLA violations pile up during a traffic spike, because the system physically cannot spin up fast enough to absorb the sudden demand. Industry analyses have found that most organizations report data and storage bottlenecks are actively limiting their AI performance and ability to scale, and most enterprises admit their data isn't AI-ready in the first place. The cold-start chain is a concrete, measurable symptom of that larger readiness problem.

F|Stage one: container image pull and peer-to-peer prefetch versus registry throughput

The image pull is usually where the pain starts, and at fleet scale it's often the worst offender, because dozens or hundreds of nodes try to pull from the same central registry at the same instant, saturating network and disk I/O all at once.

The sizes involved don't help. LLM serving containers routinely run 10 to 15 GB once CUDA libraries and framework dependencies get bundled in, and training images typically clear 25 GB. K|Pulling those at fleet scale creates a straggler problem: one slow node holds up an entire job while everyone else waits. The obvious fix, a faster registry or bigger network pipes, doesn't actually solve anything. It speeds up a single origin serving one client, but the real problem is concurrency: every node hitting that same origin at once. Wider pipes just mean a wider traffic jam.

BootSeer's answer is more surgical. Instead of pulling the whole image, it identifies the small, stable set of "hot blocks" (the data actually needed to get a job started) and distributes just those via peer-to-peer sharing between worker nodes, sidestepping registry contention entirely. The system also adds dependency snapshotting, so runtime dependencies don't get reinstalled from scratch every time, plus striped HDFS-FUSE access for checkpoints. Deployed on real production LLM training workloads, BootSeer cut total startup overhead by roughly half.

A|Baking the model straight into the container image is a related option. Google's Cloud Run documentation confirms there's no hard cap on total image size, though individual layers are capped at 9.9 GB when pulling from Docker Hub or an external Artifact Registry remote repo, and Google itself warns that the build process bottlenecks on network throughput past a certain point. Google's own recommendation above that threshold is to download weights from object storage at startup instead of embedding them in the image. Either way, the lesson from this stage is the same: treat the pull as a data-distribution puzzle (which blocks, shared how, to how many nodes) rather than a bandwidth problem you can throw money at.

Stage two: model weight loading, where storage architecture is the deciding variable

B|Weight loading is where S3, a convenient source of truth, becomes a direct tax on latency. O|Storage architecture, not inference engine tuning, sets the actual time floor at this stage.

The numbers explain why this stage carries so much weight, no pun intended. A large frontier model can carry hundreds of gigabytes of weights; even a modest model still needs tens of gigabytes. S|A large frontier model carries hundreds of gigabytes of weights, and even a smaller model still demands tens of gigabytes; these bytes must travel from wherever they're staged into GPU VRAM, and the path, a shared filesystem, node-local NVMe, or peer-to-peer transfer, sets the floor on how fast that can happen. One fix works at the engine level: rather than waiting for the entire model to land before serving starts, modern runtimes use lazy loading or weight streaming, so the engine starts generating a first token as soon as the earliest layers are sitting in VRAM.

The structural fix works at the storage layer. Layer storage into tiers: NVMe hot, S3 warm, archival cold. Keep the weights actually in use on NVMe-backed storage with sub-millisecond latency, while S3 stays the long-term source of truth for the full model corpus. NVMe-over-Fabrics and RDMA make this practical, moving data from flash media straight to GPU memory at sub-millisecond latency, and the Neural Base storage I/O analysis treats this as current best practice.

A handful of infrastructure patterns now aim at exactly this gap. A cloud-agnostic data layer that mounts directly on existing S3, GCS, R2, or Azure Blob buckets, with NVMe caching sitting in front and no ETL pipeline required, removes the false choice between "S3 as source of truth" and "fast weight loading". Archil is one option built around this idea: weight loading is the stage where the choice of storage architecture, not inference engine tuning, determines how long this takes. AWS took a step in a similar direction with Amazon S3 Files, generally available since April 7, 2026, which mounts S3 buckets as NFS file systems so workloads can hit object storage without touching their code, closing part of the S3-to-GPU gap for teams that can't swap out their storage backend. The pattern across all of these is the same: stop building custom pipelines to shuffle weights around before the job can start, and treat the filesystem itself as the interface.

Stage three: CUDA initialization, the CPU-bound phase that faster GPUs cannot fix

CUDA context setup, KV cache allocation, and CUDA graph compilation are predominantly CPU-bound, so provisioning a faster GPU does not shorten this stage. Buying a faster GPU does nothing for this stage, because the GPU isn't what's slow.

The vLLM "Breaking the Ice" paper (Kabakibo, Trivedi, Wang, MLSys 2026, Bellevue) establishes that engine and CUDA context setup in vLLM is predominantly CPU-bound, with consistent and interpretable scaling trends with respect to model- and system-level parameters rather than GPU-level parameters. That finding reframes the whole problem: this stage needs a different kind of fix than the storage-layer work in stages one and two.

Snapshot and restore is that fix. If the fully initialized state of a running inference worker, both CPU-side and GPU-side, can be serialized once and restored on demand, the entire CUDA initialization phase can be skipped on every subsequent start. Serverless computing already proved this works elsewhere: AWS Lambda SnapStart snapshots a Firecracker microVM right after its INIT code finishes running, caches that snapshot across three tiers (on the worker itself, across a placement group, and in S3 at the regional level), then restores from the snapshot instead of re-running initialization from scratch. The results are dramatic: SnapStart takes Python functions that used to cold-start in several seconds down to a few hundred milliseconds. Modal applies the same idea specifically to GPU workloads, and its GPU Memory Snapshots feature cuts vLLM startup from 45 seconds down to a matter of seconds.

None of this comes free, though. Serializing KV cache state across multiple workers opens up failure modes that don't have clean answers yet: correctness, fairness between requests, isolation between tenants, and rules for when old snapshot state should get deleted all become murky once temporary model state turns into a long-lived resource shared across machines. That's a real operational risk, and teams adopting Dynamo Snapshot or similar systems need to design around it explicitly rather than treat it as a solved problem.

Stage four: service registration and traffic routing, where orchestration reduces wasted re-initialization

Loading the weights and finishing CUDA setup still isn't the finish line. A worker can't serve a single request until service discovery registers it and the router starts sending it traffic, and in systems that are KV-cache-aware, the routing decision either shares the cold-start cost efficiently or forces the very next request to pay it all over again.

NVIDIA's Dynamo orchestration layer targets this stage with two mechanisms working together. NIXL, the NVIDIA Inference Xfer Library, moves data GPU-to-GPU at high speed, letting an already-warm node hand its state directly to a freshly provisioned one, so the new worker never has to reconstruct its KV cache from nothing. KV-Aware Routing complements this by routing requests based on worker load and how much KV cache overlap exists, so a request can land on a GPU that already holds the relevant prefix and skip redundant prefill work entirely. Dynamo's ModelExpress component streams model weights over NIXL and NVLink, and Dynamo's own repository reports roughly 7x faster model startup for DeepSeek-V3 on H200 hardware compared to the baseline.

A different, complementary approach skips reacting to cold starts altogether and tries to predict them instead. SLOPE, developed at the Athens University of Economics and Business, is a prediction framework for FaaS platforms that trains a neural network on historical run data to estimate how many instances a workload will need to meet its invocation rate, keeping a predictively sized pool warm rather than scrambling after the fact. For traffic patterns that are reasonably predictable, this kind of forecasting can eliminate cold starts almost entirely, without paying for always-on capacity around the clock. ColdStartNet, presented at ICCCN 2025 and published by Springer, takes a similar predictive approach using a hybrid stack of Random Forest, XGBoost, LightGBM, ARIMA, and LSTM models to forecast cold-start latency minutes in advance, giving infrastructure teams enough lead time to allocate resources before the spike actually hits.

K|Put the four stages together and the pattern holds across all of them: image pull, weight loading, CUDA init, and traffic routing each fail for a different reason, so each one needs its own fix rather than a single silver bullet [3][1]. Fleets that treat cold start as one undifferentiated delay end up throwing bandwidth or bigger GPUs at problems that were never about bandwidth or GPU power in the first place.