A file of a few kilobytes costs roughly the same metadata work as a file of a few gigabytes: a create, an open, a stat, a close, an entry in a directory. When the payload is tiny, the overhead dominates completely.

This has always been true. What changed is that the dominant new workload produces millions of small files by default.

Why training pipelines do this

A dataset arrives as individual images, audio clips, or documents. Somebody extracts the archive onto shared storage because that is the obvious thing to do. The training loop then reads them in random order, repeatedly, from many workers at once.

Per epoch that is millions of opens. The bytes transferred might be modest. The operation count is enormous, and it is random, and it is concurrent.

Meanwhile the file system was benchmarked and purchased on its ability to read enormous files sequentially.

The symptoms

Accelerator utilisation sits low while the storage looks under-loaded on the bandwidth graphs. Epoch time does not improve when you add storage capacity or bandwidth. Data loader worker processes spend their time waiting. Somebody suggests the answer is more accelerators.

If accelerators are idle and bandwidth is low, you are not bandwidth bound. Look at operation rate and latency.

The fix, in order of effectiveness

Pack the data. Convert the dataset into a small number of large sequential container files, with an index, read in large blocks. Every framework has a format for this. This single change routinely transforms the workload from metadata-bound to bandwidth-bound, which is the shape your file system was designed for.

It is the highest-leverage change available and it belongs to the data science team, not the storage team. Which is why it so often does not happen: the people who feel the pain cannot make the change, and the people who can make the change do not know it is the cause.

If you take one thing from this series, take that sentence and go and have the conversation.

Cache locally. If the dataset fits on node-local NVMe, copy it once at job start and read locally thereafter. A few minutes of staging buys an entire run of local-speed access.

Fix directory structure. If you must keep individual files, do not put them all in one directory. Hash into a tree. Every file system on earth handles a thousand directories of a thousand files better than one directory of a million.

Then tune the storage. Metadata on fast media, distributed namespace, relaxed client caching where semantics allow. Real gains, smaller than the ones above.

What not to do

Do not buy a bigger file system to solve it. The problem is operation rate, not capacity or bandwidth, and adding storage servers does not add metadata capacity in most architectures.

Do not assume flash fixes it. Flash lowers the per-operation latency, which helps, and leaves the operation count untouched, which is the actual problem.

The organisational version

This is a storage problem caused by a data engineering decision, diagnosed by a storage team, fixable only by a machine learning team, and visible in a metric — accelerator utilisation — that belongs to neither.

The sites that handle it well have somebody whose job spans the boundary. The sites that do not have an expensive cluster running at a fraction of its capacity while both teams believe the other one is the problem.

Next: the failure modes nobody warns you about.