The headline utilisation figure from accelerator monitoring tools is the one that ends up on dashboards, in capacity reports, and in arguments about whether more hardware is needed. It is measuring something narrower than people assume.
What it actually reports
The standard utilisation figure is, roughly, the proportion of time during which at least one kernel was executing on the device. It is a measure of occupancy over time, not of how much of the device’s capability was in use.
A single tiny kernel, using a fraction of the compute units, running continuously, reports as fully utilised. A device reading one value in a loop reports as busy.
So the metric answers “was the device doing something” and not “was the device doing much”. Those are very different questions when the hardware costs what this hardware costs.
The numbers that mean more
Achieved occupancy and streaming multiprocessor efficiency. How much of the parallel capacity is actually engaged.
Memory bandwidth utilisation. A great many machine learning workloads are memory bound rather than compute bound, which means the compute units are waiting for data. If bandwidth is saturated and compute is not, buying a device with more compute will not help you.
Tensor core utilisation, where applicable. If your workload is not hitting the specialised units, you are paying for silicon you are not using, and the cause is frequently a precision or library configuration issue rather than anything fundamental.
Power draw. Crude and useful. A device doing heavy work draws near its limit. A device reporting full utilisation while drawing a third of its rated power is not working hard.
That last one is the quickest sanity check available and requires no instrumentation beyond what is already there.
The pattern to look for
High reported utilisation, low power draw, low memory bandwidth. That combination means the device is busy with small work and the bottleneck is somewhere else: data loading, preprocessing on the CPU, synchronisation between nodes, or a batch size too small to fill the device.
I have seen a cluster reported as fully utilised where the actual constraint was a data loader that could not keep up. The dashboard said the accelerators were saturated. They were saturated with waiting.
What to do about it
Profile once, properly. Run the real workload under a profiler and find out where time goes. An afternoon of this is worth more than months of dashboard watching.
Check the input pipeline. If accelerators are idle between batches, the problem is upstream. More workers, prefetching, packed data formats, local caching.
Check batch size. Too small and the device cannot be filled. This is the single most common cause of poor efficiency on a correctly built cluster.
Check precision. If you are not using the reduced-precision paths the hardware provides, you are leaving a large multiple on the table.
Why this matters commercially
Capacity decisions get made on these numbers. A report showing 90% utilisation is used to justify buying more accelerators. If that 90% is occupancy rather than efficiency, the money would be better spent on the data pipeline, and the new hardware will be 90% utilised too.
Before approving an expansion, ask for power draw and memory bandwidth alongside utilisation. If the answer is that nobody collects those, that is the first finding, and it is cheaper to fix than a hardware purchase.