Requests arrive as “we need two petabytes”. Capacity is the easiest requirement to state and the least informative. Here is the order I work in.

1. Aggregate read bandwidth at peak

How much data must flow to all clients at once, during the busiest phase?

For training clusters the arithmetic starts with keeping accelerators fed. Take the per-node ingest rate the workload needs, multiply by nodes, and you have a floor. If nobody knows the per-node rate, measure it on one node with real data before designing anything.

This number, not capacity, determines how many storage servers you need.

2. Write bandwidth at peak, which is usually a checkpoint

Writes are typically bursty and dominated by checkpointing. This has its own post; the short version is that checkpoint bandwidth is the aggregate memory being written divided by the time you will accept for it, and it is often several times the read requirement.

3. Metadata operations per second

Files touched per job, times concurrent jobs, divided by acceptable startup time. Crude, and vastly better than omitting it. If your workload is millions of small files, this number will size your metadata tier and possibly the whole design.

4. Capacity, finally

Now capacity, and three separate figures: the active dataset, the retained dataset, and growth over the term. They are frequently different by an order of magnitude, and conflating them produces either an expensive flash system holding cold data or a slow system holding hot data.

5. Back-calculate the servers

Given per-server throughput from the vendor, divide your bandwidth requirement by it, add failure headroom, and compare against the server count implied by capacity. Take the larger.

This is the step where the design gets interesting, because on flash the two answers are often far apart, and the gap is the conversation about tiering.

6. Check the network, twice

Client side: can a single client’s network interface carry its share? A node with one 100 Gb link cannot ingest more than roughly 12 GB/s no matter what you build.

Server side: can each storage server’s links carry what its media can deliver? Flash servers routinely have more media bandwidth than network bandwidth, and the specification sheet quotes the media.

I have seen more file system designs undone by this step than by any other. Check it explicitly and write the number down.

7. Failure headroom, honestly

What happens when a storage server is down? On most architectures the capacity and the bandwidth it provided are both gone. If you sized to exactly your requirement, you are now below it during every maintenance window and every failure.

Size for N+1 at minimum, and state which N you mean.

The numbers to demand from a vendor

  • Throughput per storage server, read and write, with the block size stated
  • Metadata operations per second, with the operation mix stated
  • Rebuild time for a failed device at your capacity point, and the performance impact during it
  • Behaviour with one server down: what is the degraded throughput

That last one is rarely volunteered and is always interesting.

The most common sizing error

Sizing entirely from capacity, because capacity is what the customer said and capacity is what the price list is organised around. The result is a system with adequate space and inadequate bandwidth, which reads to users as “the storage is slow” forever.

The second most common is sizing bandwidth from a benchmark that used large sequential I/O, for a workload that is nothing like it. Always ask what the benchmark did. If it was one enormous file read sequentially and your workload is four million images, the number is not relevant to you.

Next: checkpointing, and why it often sets the whole design.