A generation of parallel file system design assumptions came from spinning disks: sequential access is fast, random access is slow, so lay data out sequentially and stripe widely to aggregate spindles. Flash changes the premise, and a newer group of systems is built on the new one.
What genuinely changed
Random access stopped being a catastrophe. The read-modify-write penalty for misaligned I/O is far less punishing. Workloads that were unthinkable on disk-backed systems — random reads across millions of small objects, which is to say machine learning training — became viable.
Small files got better. Not perfect, but the latency floor dropped by orders of magnitude, and a lot of the pain in this area was seek time.
Density. Petabytes in a rack rather than several. For sites where power and floor space are the binding constraint, which is increasingly all of them, this matters more than the performance.
Rebuild times. Rebuilding a failed device is dramatically faster, which shortens the window of reduced redundancy. This is an availability improvement that rarely appears on the datasheet.
What did not change
Metadata is still the bottleneck. Faster media makes each metadata operation quicker. It does not make a single metadata server handle an unbounded rate, and it does not fix a data layout with four million files in one directory. The cliff moved; it did not disappear.
The network is still the ceiling. When the media stops being the slow part, the fabric becomes the slow part. A client with a 100 Gb link cannot exceed roughly 12 GB/s regardless of what sits behind it. Flash systems routinely expose network design flaws that disk systems hid.
Layout still matters. Striping decisions still determine whether one file can be read at aggregate speed. The penalty for getting it wrong is smaller in absolute latency and identical in shape.
You still have to think about the client. Client-side caching, buffer sizes, and the number of outstanding requests determine whether a client can actually consume what the system can deliver. On flash systems the client is frequently the limiter, and nobody looks there.
The architectural divergence
The newer flash-native systems make a different bet: disaggregate compute from storage entirely, put all state in NVMe accessible over the fabric, and let stateless front-end servers handle protocol. This is a real architectural difference rather than the same design with faster drives, and it delivers genuinely different metadata behaviour.
The trade-off is commercial. These are proprietary systems sold at proprietary prices, on a capacity model, and the cost per terabyte is not comparable to a self-assembled Lustre deployment. Whether that is worth it comes down to the same question as always: what is your operational capacity worth, and what does an hour of idle GPU cost you.
The sizing consequence
On disk-based systems, capacity and bandwidth were coupled: you added spindles for both, and you frequently ended up with more capacity than you needed because you needed the spindles.
On flash, they decouple. You can buy the bandwidth you need in far less capacity, or the capacity you need with bandwidth to spare. This means you must size for both explicitly rather than assuming one drags the other along.
It also means the old habit of over-buying capacity to get performance is now visible as waste rather than hidden as necessity, which makes the conversation with finance sharper in both directions.
The one-line version
Flash removed a physical constraint and exposed every logical one behind it. The systems are better and the design work is less forgiving, because the bottleneck moved somewhere you cannot fix with a purchase order.
Next: how to actually size one of these things.