A single-server file system fails in a way everyone understands: it is up or it is down. Distributed systems fail in a wider variety of ways, most of which are partial, and partial failure is harder to diagnose than total failure.
The slow component
One storage target is degraded — a failing drive, a rebuild, a saturated link. It still serves requests, just slowly.
Every file with a piece on that target is now slow. Those files are scattered across the namespace with no pattern a user could perceive. Reports arrive as “sometimes it is slow”, which is the least actionable sentence in operations.
Detection requires per-target latency monitoring. Build it before you need it. Without it you are correlating user anecdotes against a system with no per-component visibility, which is not a diagnosis, it is a guess.
The noisy neighbour
One job on one node hammers the metadata servers or saturates a target. Everyone else’s interactive work becomes unpleasant. The offending job may be perfectly legitimate.
Mitigations exist — request rate limiting, quality of service, per-user throttling — and are unevenly implemented across systems and frequently switched off because they were not configured at install. Check whether yours has them and whether they are on.
The client that will not let go
A client holds a lock on a file, then hangs, swaps, or loses the network. Other clients wanting that file wait. Depending on the system, they wait a long time before anything times out.
The symptom is a hang affecting a subset of users on a subset of files. The cause is on a machine that is not the storage. This is why parallel file system incidents so often require looking at compute nodes, and why “the storage team” cannot resolve them alone.
Cascading eviction
A server under load stops responding to clients in time. It evicts them. Evicted clients reconnect, all at once, generating a load spike that causes more timeouts.
Once this starts it can be self-sustaining. Breaking it usually means reducing load, which means stopping jobs, which means someone has to decide whose work dies.
The prevention is capacity headroom. A system running near its limit has no slack to absorb a transient, and the transient becomes an incident.
Metadata corruption
Rare, serious, and the reason the metadata tier is mirrored and backed up. If the namespace is damaged, the data on the storage targets may be intact and unreachable, which is functionally identical to lost.
Verify that the metadata backup exists and has been restored in a test. Not verified means not present.
Split brain and quorum
In systems where nodes vote, a network partition can leave two groups each believing they are authoritative. Good systems prevent this with quorum rules. Those rules depend on a sensible node role design, which depends on somebody having thought about it at install time.
If your cluster design was copied from a reference architecture without adjusting for your network topology, this is worth revisiting before it is tested by an outage.
The upgrade
Not a failure mode exactly, but the most common cause of unplanned downtime in practice. Version compatibility between clients and servers is a real constraint, kernel dependencies are real, and a partially upgraded cluster can behave strangely.
Treat upgrades as projects with a rollback plan. Sites that defer upgrades because they are painful end up on unsupported versions, which makes the eventual upgrade worse and removes the option of vendor help during an incident.
What to build on day one
Per-target throughput and latency. Metadata operation rate. Client count and eviction events. Capacity per target, because imbalance is itself a problem. Alerting on any target deviating from its peers.
That last one is the highest-value alert in the whole system. A target that is behaving differently from its siblings is the earliest signal you will get, and it usually arrives before any user notices.
Next: why you cannot back up one of these the way you think.