The benchmark says 200 GB/s. The users say the file system is slow. Both are true, and the gap between them is metadata.

What counts as metadata work

Opening a file. Creating one. Deleting one. Listing a directory. Checking a file’s size or permissions. Anything that touches the namespace rather than the contents.

These operations are small, numerous, and frequently serialised. A job that reads one enormous file uses almost no metadata capacity. A job that opens four million small files uses nothing but metadata capacity, and the storage servers sit idle while the metadata servers melt.

Why AI made this worse

Traditional HPC simulation was kind to metadata: a few large files, written sequentially, read sequentially. The workloads that pushed these systems were bandwidth workloads, and the systems were designed for them.

Machine learning training is the opposite. A dataset of millions of individual images or text shards, read in random order, repeatedly, by many workers at once. Every epoch is millions of file opens. The bandwidth requirement may be modest while the metadata requirement is brutal.

Organisations buy a file system benchmarked on the first workload and run the second on it. The disappointment is structural, not a configuration error.

The symptoms

You are metadata bound if:

  • ls on a large directory takes seconds or minutes
  • Job startup is slow while the job itself runs fine once going
  • Throughput graphs show the storage servers under-utilised during a slow period
  • Adding more storage capacity changed nothing
  • One user’s workload makes everyone else’s interactive work unpleasant

That last one is the political symptom. One person untarring a large archive can degrade the namespace for the entire cluster, and it will be reported as “the storage is down”.

What actually helps

Fix the data layout first. Millions of small files in one directory is the worst case for every system ever built. Hashed subdirectory trees help enormously. Packing training data into larger container formats — sharded archives read sequentially — helps more than any hardware change, and it is a change to the data pipeline rather than the storage.

I keep coming back to this because it is the highest-leverage fix and it is nobody’s job. The storage team cannot change how the data is written. The ML team does not know the storage is why their epochs are slow.

Put metadata on fast media and enough of it. Metadata targets should be NVMe, mirrored, with generous memory on the servers. This is not where to save money; it is a small fraction of the capacity and a large fraction of the experience.

Distribute the namespace. Most systems now support spreading directories across multiple metadata targets. Use it. A single metadata server is a single bottleneck no matter how fast it is.

Cache aggressively on clients where semantics allow. Many workloads do not need strict coherence on every stat call, and relaxing that is a supported option in most systems.

Sizing for it

Ask for a metadata operations per second figure alongside the bandwidth figure, and make the vendor state which operation mix it was measured with. Create-heavy and stat-heavy workloads produce very different numbers on the same hardware.

Then estimate your own: number of files touched per job, multiplied by concurrent jobs, divided by acceptable startup time. It is a rough number and it is infinitely better than not having one.

The thing to remember

Bandwidth is what gets sold because bandwidth is what benchmarks well and what fits on a slide. Metadata is what determines whether the people using the system think it is good. If you take one number into a vendor conversation, take that one.

Next: striping, and the defaults that quietly cost you most of your throughput.