This is the first of a series on parallel file systems: how they work, how to size them, and what goes wrong with them in the field. I have been spending more time on this because AI infrastructure has dragged a technology that lived in national laboratories into ordinary enterprise data centres, and the operational knowledge has not travelled with it.
The one-sentence version
In a conventional network file system, a client talks to one server, and that server is the bottleneck for that client. In a parallel file system, a single client reads and writes one file across many storage servers simultaneously, and aggregate bandwidth scales with the number of servers rather than the capability of one.
Everything else follows from that sentence.
The three roles
Almost every parallel file system separates three jobs, whatever it calls them.
Metadata. Who owns this file, how big is it, when was it modified, which storage targets hold its pieces. One or more dedicated servers handle this. In Lustre these are the MDS and MDT; in IBM Storage Scale the function is distributed differently but the job is the same.
Data. The actual file contents, spread across many storage servers. Lustre calls them OSS and OST. Adding more of these is how you add both capacity and bandwidth.
Client. Not a bystander. The parallel file system client is a kernel module or driver that knows the layout, computes which servers hold which pieces, and issues parallel requests. This is the crucial difference from NFS: the intelligence is on the client.
Striping is the mechanism
A file is cut into chunks and those chunks are distributed across storage targets in a pattern. Two parameters govern it: how many targets the file is spread across, and how large each chunk is.
Set these well and a single large file can be read at the aggregate speed of dozens of servers. Set them badly and you get a file system that cost a fortune and performs like one disk. This gets its own post, because the defaults are wrong for most workloads and almost nobody changes them.
Why it is not just a faster NAS
Three consequences that catch people.
The client is part of the system. Client software version, kernel version, and network configuration all affect correctness and performance. You cannot treat the compute nodes as opaque consumers. An upgrade is a coordinated event across the whole cluster.
POSIX semantics cost something. Maintaining the illusion that many machines writing to one file system see a consistent view requires locking, and lock contention is the source of a large fraction of real-world performance complaints.
Failure is partial and visible. When one storage target is slow, every file striped across it is slow. The blast radius of a single degraded component is wider than in a system where files live in one place.
When you need one
Honestly: when aggregate bandwidth to a shared dataset is the constraint, and no single server can supply it. That is HPC simulation, large-scale training, and genomics. It is not most enterprise file serving, and a parallel file system deployed where a good scale-out NAS would do is an expensive way to acquire operational complexity.
The test I use: does a single job need more read bandwidth than one storage server can deliver, and does it need it against data that other nodes are also using? If yes, you are in this territory. If the answer is “we have a lot of data”, you are not — that is a capacity problem, and capacity problems have simpler solutions.
Next: why metadata, not bandwidth, is where these systems actually fall over.