The first question in any accelerated sizing is whether the model fits. The arithmetic is simple enough to do on a napkin, and the napkin version is wrong in a specific and predictable direction.
The weights
Parameters multiplied by bytes per parameter.
At 16-bit precision, two bytes each. At 8-bit, one. At 4-bit, half. So a model of a given parameter count needs roughly twice its parameter count in gigabytes at 16-bit, once at 8-bit, half at 4-bit.
That is the number everyone computes, and it is the floor, not the requirement.
The key-value cache
For inference, this is the part that gets forgotten and the part that determines your actual capacity.
Every token in a request’s context occupies cache space, proportional to the number of layers, the attention head dimensions, and the precision. It scales with sequence length and with the number of concurrent requests.
The practical consequence: a model that loads comfortably will run out of memory under concurrency, at a point determined by how long your contexts are. Doubling the context length doubles the cache per request. Doubling concurrency doubles it again.
If you size for weights alone, you will discover your real concurrency limit in production.
Activations and overhead
Working memory for the forward pass, framework overhead, fragmentation, the runtime’s own allocations. Budget something meaningful for this rather than assuming the remainder is usable.
Training is a different calculation entirely
For training, weights are the small part. You also need gradients, optimiser state, and stored activations for the backward pass. Depending on the optimiser and what you checkpoint, total memory can be several times the model size — commonly quoted as a multiple in the region of four to six times for full fine-tuning with a standard optimiser at mixed precision.
This is why full fine-tuning of a model that runs happily for inference on one device needs several, and why parameter-efficient methods that train a small adapter instead are so widely used: they cut the memory requirement dramatically and often produce a perfectly good result.
If somebody tells you they will fine-tune a model that just fits in memory for inference, on the same hardware, ask which method. The answer is either an efficient one or a misunderstanding.
Quantisation as a capacity decision
Dropping precision is the main lever. From 16-bit to 8-bit is close to free in quality for most use. From 8-bit to 4-bit is noticeable on hard reasoning and invisible on summarisation and extraction.
Treat it as a capacity decision with a quality cost, not as a quality decision. The question is what accuracy you need for the task, tested on your task, not on a benchmark.
Multiple devices
When it does not fit, you split. Splitting by layer across devices is simple and leaves devices idle in turn. Splitting each layer across devices keeps them all busy and requires fast interconnect, because they exchange data at every layer.
Which means: if your sizing conclusion is “we will use two devices”, check what connects them. Over a slow link, layer-splitting across two devices can be slower than a smaller model on one.
The sizing I would actually do
Take the model, the precision you intend, the maximum context length you will support, and the concurrency you need at peak. Compute weights plus cache at that concurrency, add a third for overhead, and compare with the device memory.
Then test it, because the arithmetic gives you a plan and the runtime gives you the truth.