A customer asks for a quote for an eight-GPU node. What they mean is “I want to run a model”. What they will receive is an education in facilities engineering, and it is better if that arrives before the purchase order rather than after.
Here is the list I now walk through, in the order the surprises arrive.
Power, per rack, not per server
A dense accelerated node draws something in the region of 6 to 10 kW under load depending on the part. Put four in a rack and you are asking for 30 to 40 kW to a single rack position. Many enterprise data halls, including several very good ones I have stood in, are provisioned around 5 to 8 kW per rack.
This is not a problem you solve with a bigger PDU. It is a problem you solve by spreading four servers across four racks, which wastes rack space, or by upgrading a power distribution design, which is an electrical project with a lead time measured in quarters.
Ask for the actual per-rack power budget, in writing, from the facilities team. Not the design figure from the original build. The current figure, accounting for what is already installed.
Cooling, and the moment air stops being enough
Air cooling works up to a point and then it very clearly does not. The point moves with each hardware generation, and it has been moving downwards in terms of what air can absorb per rack. Once you cross it you are talking about rear-door heat exchangers or direct liquid cooling, and both are building decisions, not IT decisions.
Liquid brings a new category of question that IT teams have never had to answer: who maintains the loop, what happens on a leak, what is the water quality requirement, and who is on call for it at 3am. I have watched a project stall for four months on the last one.
Floor loading
Rarely checked, occasionally fatal. Dense racks are heavy. Raised floors have limits. Somebody with a structural engineering qualification should confirm this, once, in writing, and then everyone can forget about it forever.
The fabric
This deserves its own post and will get one. In short: for anything involving more than one node cooperating on a single job, the interconnect is not a networking detail, it is part of the compute design. Under-provision it and you have bought expensive accelerators that spend their time waiting for each other.
Budget for it explicitly. The switching and cabling on a serious training cluster is not a rounding error against the server cost.
Storage, and the read pattern nobody described
Training reads datasets repeatedly at high concurrency. Inference reads model weights once and then mostly does not touch storage. Fine-tuning sits between. These are three different storage problems and people ask for one storage solution.
The failure mode I see most often is a general-purpose array being asked to feed accelerators, achieving it on paper, and falling over on the concurrency. Ask what the read pattern actually is. If the answer is vague, the honest position is that you cannot size storage yet.
Lead time
Say it early and say it plainly: the accelerators have a lead time, and it is not two weeks. Plan the facilities work to complete before the hardware lands, because the reverse order means expensive equipment sitting in a crate while an electrician is scheduled.
The part that is actually hard
None of the above is the difficult conversation. The difficult conversation is this: the requirement is usually stated as a number of GPUs, and a number of GPUs is an answer, not a requirement.
The requirement is a model size, a number of concurrent users, an acceptable latency, and a utilisation expectation. From those you can derive hardware. Without them you are buying capacity by intuition, and intuition in this area is currently calibrated by press releases.
I ask four questions and I will not size without them:
- Which models, at which parameter count and which precision?
- Training, fine-tuning, or inference — and what is the honest split?
- How many concurrent requests at peak, and what latency is acceptable?
- What happens to this cluster in eighteen months?
The fourth is the one people find odd and it is the one that most often changes the design. A cluster bought for a proof of concept and a cluster bought as the first tranche of a platform are different machines, even when the proof of concept is identical.
Where this leaves you
Accelerated infrastructure is the first thing in twenty years that has made the facilities team a mandatory participant in an IT design review. That is not a complaint. It is a fairly significant change in how these projects have to be run, and the organisations handling it well are the ones who noticed early and put the right people in the room.