Lustre powers a very large share of the world’s fastest systems and it is genuinely fast. None of the complaints below are about throughput. They are about what it is like to run.

The client is part of the system

The Lustre client is a kernel module. That means it is tied to kernel versions, which means an operating system upgrade on your compute nodes is a Lustre compatibility question. Mismatched client and server versions are supported only within defined ranges.

Consequence: upgrades are cluster-wide coordinated events, not rolling maintenance. Plan them as projects. Sites that fall behind on versions do so because the upgrade is disruptive, and then they are running something old, which makes the next upgrade worse.

Client eviction

When a server decides a client is unresponsive, it evicts it. The client’s in-flight operations fail, and depending on the application, the job dies.

Evictions are usually a symptom of something else: network trouble, a server under extreme load, a client that swapped or hung. But the failure presents to the user as “my job crashed and the storage is broken”, and diagnosing it means correlating logs across clients and servers at the timestamp in question.

Learning to read these logs is the single most valuable operational skill for a Lustre site, and it is not something you can acquire during an incident.

One slow target degrades everything striped across it

Because files are spread across targets, a single target that is slow — a failing drive, a rebuild in progress, a saturated server — slows every file that has a piece on it. The system does not fail; it degrades, broadly and confusingly.

Users report that “the file system is slow” with no pattern, because the affected files are scattered. Finding the sick target requires per-target monitoring, which is why per-target monitoring is not optional.

This is the thing I would tell anyone deploying: build the dashboard that shows latency and throughput per storage target, on day one. Without it you are diagnosing a distributed system by anecdote.

The small-file penalty is real and permanent

Lustre is built for large sequential I/O. It handles small files, but not well, and no amount of tuning makes it a good small-file system. Sites that discover this after loading a few hundred million small files are facing a data layout migration, which is far more painful than the original decision would have been.

Quotas and accounting

Functional, and fiddly. Multi-tenant sites in particular find that the quota model does not map neatly onto how they want to bill or limit. Worth testing against your actual policy before you promise anything to your users.

The skills question

Lustre assumes you have somebody. Not a generalist who read the documentation once, but a person who understands the architecture and can read the logs. In national labs and large HPC centres this person exists and is excellent. In an enterprise that bought an AI cluster last year, they frequently do not.

This is the honest decision point. Lustre is free software with commercial support options, and the software cost saving is real. If you do not have the person, that saving is illusory, because you will either buy the expertise as a service or spend it in outages.

The alternatives — commercially supported parallel file systems, or appliance-delivered ones — cost more in licensing and less in expertise. Which is cheaper depends entirely on what your organisation actually has.

Next: IBM Storage Scale, which solves some of this and introduces a licensing conversation instead.