Enterprise networks were designed for a traffic pattern where most conversations are short, most flows go north-south to a client or a gateway, and oversubscription in the aggregation layer is fine because the odds of everything talking at once are low.

A distributed training job violates every one of those assumptions simultaneously. It is a small number of enormous flows, entirely east-west, that all start at the same instant because they are synchronised by the algorithm. Every node finishes its step and then every node needs to exchange gradients with every other node, at once, repeatedly, for days.

If you run that pattern over a fabric built on enterprise assumptions, the job completes. It just completes much more slowly than the hardware datasheet suggested, and the people who bought the accelerators will assume the accelerators are the problem.

Why oversubscription stops working

An oversubscribed leaf-spine design is a bet that not every port needs its full bandwidth at the same moment. The collective operations at the heart of distributed training are specifically a moment when every port needs its full bandwidth at the same moment.

For clusters that matter, the design target is non-blocking, or as close as budget allows. When you cannot afford non-blocking, at least know your ratio and expect the job time to reflect it.

The tail is the metric, not the average

The other assumption worth discarding is that average latency is the number to watch. A synchronised collective finishes when the slowest participant finishes. One congested path, one retransmission, one buffer that filled at the wrong moment, and the whole step waits.

So the metric that predicts cluster performance is tail latency and packet loss, not average throughput. A fabric can show excellent averages and be terrible for this workload.

Lossless is a configuration project

RDMA over converged Ethernet needs the fabric not to drop frames, which means priority flow control and explicit congestion notification configured consistently end to end: every switch, every NIC, every hop. Configured consistently is doing a lot of work in that sentence.

The failure mode is not an error message. It is a cluster that works, passes a basic test, and then performs at seventy percent of expectation during real jobs, for reasons that require someone to read switch counters to diagnose. Plan for a tuning phase. Put it in the schedule with a name and an owner.

Separate the networks

Three kinds of traffic want to exist in an AI cluster: the compute fabric, storage, and ordinary management and client access. Putting them on one fabric is possible and is a reliable way to make the storage traffic interfere with the collective at exactly the wrong moment.

Physical separation is simplest. Where that is not affordable, at minimum separate by priority class and be honest that you have created a tuning problem.

Cables are a real risk

I did not expect to write this sentence in 2026, but optics and cabling are a genuine source of cluster instability. High-speed links are less forgiving than the ones your team is used to. Bend radius matters. Transceiver compatibility matters. A single marginal link degrades a collective across the whole cluster, and finding it is tedious.

Buy a small number of spares. Label everything. Test every link at full rate at commissioning, not at first job.

What I ask before quoting a fabric

  • How many nodes will participate in a single job, at most, in eighteen months?
  • Is this one job at a time, or many tenants sharing?
  • Is inference the main workload? If so, this entire post matters much less to you.
  • Who will own the fabric configuration, and have they done lossless Ethernet before?

That last one again. Most of what goes wrong in this area is not a purchasing decision, it is an operational skills gap that nobody named at the time of purchase.

The honest summary

The accelerators are the visible cost and the fabric is the invisible one, and the ratio between them is not the ratio your intuition from enterprise networking suggests. Size the fabric as part of the compute, not as the thing you connect the compute with afterwards. If the budget cannot stretch, build a smaller cluster with a proper fabric rather than a larger one with a compromised one. The smaller cluster will finish the job first.