Unstructured Data Access Patterns for AI

Unstructured data (documents, images, audio, video, log files) never fit the tables and columns most storage systems were built around, and AI made that worse, not better. Training, inference, and agents touch this data in three completely different ways, and almost nobody's infrastructure was built to serve all three at once. Training wants a firehose, inference wants one specific glass of water fast, and agents want to wander the aisles, grab whatever looks useful, then circle back for more. Match the plumbing to the pattern and things scale; mismatch it and your GPUs sit there burning cash while they wait on a file to show up.

Unstructured data makes up most of what enterprises actually store, and yet only 16% of AI initiatives have reached enterprise scale, according to IBM's 2025 CEO Study. That gap isn't really about models being immature. It's about infrastructure that hands data to a model in the wrong shape.

How streaming reads define the throughput demands of AI training

Training eats data constantly and in huge volumes, and it does not want to stop for anything. The classic pipeline pulls raw files from object storage down to local SSD, decodes and augments them on CPU, batches them, then hands them to the GPU. Every handoff in that chain is a place things can stall.

When it stalls, you get GPU starvation: silicon that costs a fortune, sitting there idle, waiting on a file instead of doing math. Storage-bound training runs have been shown to leave GPU utilization well below its potential, while properly tuned I/O can push it into the high range. Industry analysis has consistently identified I/O wait states as a leading contributor to enterprise GPU idle time. Storage, not compute, is the leading cause of wasted accelerator capacity right now.

Run the numbers on what that actually costs. H100s run in the tens of thousands of dollars apiece, and idle time at cluster scale isn't a rounding error; it compounds into millions of dollars a year while everyone stares at the model architecture wondering what went wrong.

Training compute has scaled rapidly over recent years, but data movement hasn't kept up, and that gap creates a latency wall no amount of extra compute clears on its own. Checkpointing adds more weight on top: saving model state means writing hundreds of gigabytes at a stretch, and if that write is synchronous, every GPU in the cluster waits for it to finish. Fault tolerance turns into a tax on speed.

Three things separate infrastructure that works from infrastructure that doesn't here: sustained sequential throughput, parallel I/O across many files at once, and checkpoint writes that happen asynchronously in the background, out of the GPU's way. A general-purpose NAS, or even one very fast SSD, doesn't clear that bar once you're at training scale.

What object storage does well for training data and where it creates friction

Object storage earned its spot as the default home for training data honestly. It scales out without hitting a wall, it speaks S3 from basically any ML framework you'd name, and it takes every unstructured format without complaint: images, video, audio, logs, checkpoints, all of it.

The friction shows up in the same few spots every time, though. Small random reads carry high per-request latency, the so-called small file problem, and it bites hardest on datasets built from millions of tiny images or short text snippets. Listing millions of objects is slow, and that slowness stalls indexing and shuffle operations before training even starts. Cross-region setups tack on egress fees that scale right along with your data.

Purpose-built AI object storage is starting to chip away at this. Some providers have begun offering AI-focused object storage designed for global dataset access without egress charges, a direct swing at the cross-region penalty. On raw performance, WEKApod has put up numbers that reset expectations for what training storage is allowed to look like: 720 GB/s of throughput from eight storage nodes feeding 768 H100 GPUs, 351 GiB/s of sequential reads, GPUDirect Storage built in. Meanwhile PCIe Gen5 NVMe and NVMe over Fabrics are stretching near-local latency out to disaggregated storage pools, so a whole cluster shares fast NVMe without every node needing its own drive.

Plain S3 is a fine place to start, but it's not a finished answer for training-scale streaming reads, which is the gap Archil addresses by mounting existing buckets as a native filesystem so workloads skip the data-movement step entirely. Most teams learn that the hard way, usually by bolting on a caching layer, building a prefetch pipeline, or giving up and moving to a parallel filesystem instead.

How inference flips the access pattern from streaming to random

Inference doesn't march through a corpus in order. It reaches in and grabs one specific thing, a document, an embedding, a media file, in response to whatever a user just typed.

Retrieval-Augmented Generation is the textbook case. A query comes in, a retrieval system pulls relevant chunks out of a large unstructured corpus, and those chunks get stuffed into the prompt. Every one of those steps adds latency the user feels directly, sitting there watching a spinner.

The access pattern is random, often many requests deep at once, each one touching a different slice of data with no order to prefetch against. Metadata carries far more weight here than it does during training, since without it, retrieval leans entirely on semantic similarity, which surfaces something confidently wrong just as easily as something right. Good metadata lets a system filter by recency, source, data type, or domain before semantic ranking even starts. That's a control on accuracy, not just a shortcut for speed.

Latency tolerance flips too. A streaming read shrugs off a few hundred milliseconds of startup, amortized over a batch nobody's watching in real time. A RAG lookup that tacks a few extra seconds onto a chat response is a product complaint waiting to happen. The fix leans on low-latency caching in front of object storage, hot NVMe tiers for documents that get pulled again and again, and metadata indexes that skip a full semantic search whenever a structured filter can do the job faster.

How agents navigate unstructured data through iterative discovery rather than pre-planned queries

Agents don't know what they need until they go looking for it. They poke around, read a bit, decide what to open next, write something down, sometimes circle back to fix what they just wrote. That access pattern is iterative and self-directed, structurally different from both training (planned well ahead of time) and RAG inference (one targeted lookup).

Picture an agent working through a folder of contracts. It opens a directory listing, reads a handful of files, writes a summary, follows a reference that summary turns up, opens more files, then loops back and fixes the first summary based on what it just learned. None of that sequence got decided in advance, since the data gets found mid-task, not listed out before the task starts.

Stuffing everything into the prompt up front is a poor substitute for this kind of exploring. It forces the system to guess what the agent might need, bloats the context window with stuff that may never get used, and speeds up the point where the model starts losing the thread. Filesystem primitives, directory traversal, partial reads, in-place writes, symlinks, fit this kind of work the way a key fits one specific lock, not a master key, just the one that opens this one door.

Storage has to persist across sessions too. An agent that forgets its own progress between runs can't build on anything it did last time, which kills the entire point of working iteratively. The same working directory needs to mount cleanly across parallel agent runs, so multiple agents sharing one corpus see a consistent view instead of tripping over each other. Because agents write nearly as much as they read (intermediate files, extracted summaries, structured output), the storage layer also needs real POSIX semantics: atomic rename, proper locking. Skip that, and agents start quietly overwriting each other's work, and nobody notices until the output comes out wrong, usually days later.

Why the filesystem interface handles all three patterns when object storage APIs handle only one well

S3-style APIs are genuinely good at one thing: bulk sequential reads, which happens to be what training wants. Everywhere else, they create drag. Random small reads at inference speed need a caching layer bolted on just to feel responsive, and iterative agent work (directory listing, partial reads, in-place writes, locking) has no natural home in an object API at all. Teams end up building workarounds, or restricting what their agents are even allowed to do, which is a workaround by another name.

POSIX filesystem semantics got designed decades ago for this exact mix: open, read, write, seek, rename, lock, list. That's the whole set, and it covers all three access patterns without a translation layer sitting in between. Models trained on a huge pile of bash scripts and command-line sessions already know how to navigate a filesystem on instinct, so they don't need a bespoke object-storage SDK or some custom tool nobody's heard of. The interface they grew up on is the filesystem.

Full POSIX support isn't a nice-to-have or some philosophical stance. It's just practical: atomic rename stops partial writes from corrupting data, flock and fcntl make concurrent access safe, mmap allows zero-copy reads on big files. With that in place, existing ML code runs as-is, no rewrite required. A filesystem mounted directly on top of an existing bucket (S3, GCS, R2, Azure Blob) sidesteps the whole interface fight without moving a single byte, since the bucket stays the source of truth and the filesystem is just how you get to it.

None of this is a side issue. The AI storage market is projected to grow from $36 billion to $322 billion by 2035, a 24.4% CAGR, per MarketsandMarkets (2025). That kind of curve doesn't happen because people love buying storage; it happens because the interfaces everyone inherited were never built for any of this in the first place.

How cloud-agnostic data layers address the access problem when data is fragmented across environments

Most enterprises don't live in one cloud. They're spread across several clouds, some aging on-prem gear, and a scatter of edge locations, all at once, and every access pattern above still has to work across that entire mess, not just inside one tidy environment somebody drew on a whiteboard.

Data siloed like that leaves agents flying blind. They can only reason over what they can actually reach, and cross-cloud data gravity makes moving data to compute slow and expensive, exactly backwards from what these workloads need.

Two patterns have shown up to deal with this. Cloud-native unified layers, Google Cloud's Cross-Cloud Lakehouse with Smart Storage is one, handle automatic annotation and cross-cloud access without a custom pipeline for every source, built for agents that need to read data no matter where it lives. Filesystem-over-bucket setups solve it differently: mount a POSIX filesystem straight onto existing S3, GCS, R2, or Azure Blob storage, no migration, no ETL, no duplicate copies floating around, so the access layer stays storage-agnostic and the bucket never stops being the source of truth.

What every approach that actually works shares is this: compute moves to where the data already sits, instead of dragging the data over to compute. That single shift kills the data movement bottleneck across all three access patterns at once.

Data residency and access revocation stay the customer's job, always, no matter which pattern you pick. Any setup that keeps a persistent copy of your data sitting outside your own account is a compliance problem waiting to happen, no matter how fast it makes retrieval feel.

What matching infrastructure to access pattern looks like in practice

Stop asking what the best storage system is, and ask instead which access pattern actually runs the workload sitting in front of you right now, and what that pattern demands in return. That's the whole exercise.

For training, sustained throughput and parallel I/O matter more than shaving milliseconds off any one request. Asynchronous checkpoint writes belong in the design from day one, not bolted on once someone notices the GPUs stalling. Data should also sit close to compute, or behind a fast cache, because cross-region reads at training scale cost you twice: once in time, once in egress fees.

For inference, per-request latency is the whole game, and an NVMe caching tier in front of object storage closes most of that gap by itself. Metadata quality matters more than raw throughput here, since a dollar spent on annotation and metadata management pays back faster than a dollar spent on another storage tier.

For agents, persistent shared storage with full POSIX semantics isn't optional, it's the floor. Serverless execution sitting right next to the filesystem cuts out the round-trip of shuttling data to a separate sandbox and back. Storage metered on actual use, instead of pre-provisioned to some number somebody guessed, fits how unpredictable agent work actually is: one task needs a few megabytes, the next needs several gigabytes.

The projection that a majority of AI projects lacking AI-ready data will get abandoned through 2026 is this mismatch showing up on a spreadsheet, eventually. Those projects don't fail because the models were wrong; they fail because the data never showed up in the right shape, at the right speed, the moment the model actually needed it.

Look at each pipeline you run, and name the access pattern underneath it honestly, not the one you wish it had. Then check whether the storage and caching were actually built for that pattern, or just happened to be sitting around already.