A long-running job that fails without a checkpoint loses everything since the start. So jobs checkpoint: periodically, every participating node writes its state to shared storage, and the job pauses while it happens.

The characteristics of that write are what make it a sizing problem. It is enormous, it is simultaneous across all nodes, it is pure write, and every second it takes is a second all the compute is idle.

The arithmetic

Rough model: total state to write, divided by acceptable pause, gives required write bandwidth.

Take a cluster where each node holds tens of gigabytes of state that must be persisted, across dozens of nodes. That is a multi-terabyte write. If you will accept a sixty-second pause, you need that multi-terabyte write to land in sixty seconds, and the required aggregate write bandwidth follows directly.

Run that calculation for a realistic cluster and the number is frequently larger than the read bandwidth requirement, sometimes by several times. Which means the file system is sized by an operation that happens for one minute in every thirty.

The cost of getting it wrong

Suppose the storage can absorb half what you need. The checkpoint takes two minutes instead of one. If checkpoints happen every thirty minutes, you have lost roughly three percent of your compute time.

Three percent does not sound like much until you multiply it by what the cluster cost. On a large accelerated cluster that is a real annual number, and it is being spent on storage that was under-sized to save a smaller number.

This is the argument that wins budget for storage on AI projects, and it is an arithmetic argument rather than a technical one. Compute the idle-accelerator cost of a slow checkpoint and put it next to the price difference.

What reduces the burden

Fewer checkpoints. The frequency is a trade between restart cost and checkpoint cost. Many teams checkpoint far more often than their actual failure rate justifies, because the interval was set once and never revisited.

Asynchronous checkpointing. Write to local memory or node-local NVMe fast, then drain to shared storage in the background while compute resumes. This converts a synchronous burst into a background trickle and is the single biggest improvement available. Framework support varies; check before assuming.

Node-local scratch. A layer of NVMe in the compute nodes absorbs the burst at local speed. The shared file system then sees a gentler flow. This is what on-demand file systems across node-local flash are for.

Incremental checkpoints. Write only what changed. Effective where the framework supports it, and it is not universal.

Compression. Sometimes worthwhile, sometimes the compression cost exceeds the transfer saving. Measure rather than assume.

What to ask before designing

  • How much state does a single node hold at checkpoint time?
  • How many nodes checkpoint simultaneously?
  • How often, and who decided that interval?
  • Is checkpointing synchronous or asynchronous in this framework?
  • Is there node-local NVMe, and is it being used for this?

Five questions. The answers often reveal that the storage requirement can be cut substantially by changing the job configuration rather than the hardware, which is a better outcome for everyone except the person selling the hardware.

The point

Steady-state averages describe a system that nobody experiences. The burst describes the moment that costs money. Size for the burst, then look for ways to make the burst smaller, and do both before signing anything.

Next: GPUDirect Storage, and whether it earns its complexity.