The instinct on an AI project is to put everything on the fastest storage available, because the accelerators are expensive and nobody wants to be the reason they are idle. The result is petabytes of cold data on flash that gets read twice a year.

The tiers that actually exist

Node-local NVMe. Fastest, smallest, not shared. Ideal for the working set of a running job and for absorbing checkpoint bursts.

Shared fast tier. Parallel file system or fast NAS. Holds the active datasets that jobs read directly.

Object storage. S3-compatible, on premises or in a cloud. Cheap per terabyte, high aggregate throughput, high latency per operation, no POSIX semantics.

Archive. Tape or cold object. Cheap, slow, and where data goes when nobody has touched it in a year.

Most designs I see have two of these and would work better with three.

Where object storage genuinely fits

The raw dataset of record. The canonical copy of your training data, versioned, immutable, cheap to keep. Jobs stage what they need from it.

Output artefacts. Trained model checkpoints you want to keep, evaluation results, logs. Written once, read rarely, kept for a long time.

Anything accessed sequentially in large chunks. Object storage is good at streaming large objects and bad at small random operations. Packed dataset shards read sequentially are exactly the good case.

Where it does not

Anything expecting POSIX. Tools that want to mmap a file, or append, or rename atomically, will not work well against an object store, and gateway layers that fake POSIX over objects have sharp edges.

Small random reads at high rates. Every operation carries protocol overhead and latency measured in milliseconds rather than microseconds. Millions of small gets is a bad pattern.

The hot working set. Staging is the answer, not direct access.

The design that works

The pattern that keeps proving out: object storage holds the dataset of record, jobs stage their working set onto the fast tier or node-local NVMe at start, compute reads locally, results go back to object.

Staging costs minutes at job start and buys local-speed access for the whole run. For a job measured in hours, that trade is obviously correct, and it is frequently rejected because it feels like extra work.

The part that is actually hard

Not the technology. The policy.

Who decides what is hot? What moves data down a tier, and when? What happens when a researcher needs something that aged out last month?

Automatic tiering by access time is the usual answer and it has a failure mode: data that is accessed rarely but urgently gets demoted, and the urgent access is then slow. Manual tiering has a different failure mode: nobody does it, and the fast tier fills.

The workable middle is policy by dataset rather than by file, decided by the people who own the data, revisited quarterly. Which means somebody has to own the conversation, and on most AI projects nobody does until the fast tier is full.

Sizing consequence

If you adopt this pattern, the fast tier is sized for the active working set plus headroom, not for the total dataset. That is often a factor of five or ten smaller, and it is where the money is saved.

If you do not adopt it, you are buying the fast tier at total dataset size, and the difference will pay for a lot of object storage.

Decide which you are doing before you buy, because the two designs are not the same purchase.