Accelerators are expensive and most workloads do not use one continuously. Sharing is obviously desirable. The mechanisms available differ in ways that matter, and the differences are about isolation rather than performance.
Time-slicing
The scheduler gives each consumer the whole device in turn. Simple, works everywhere, and requires no special hardware support.
What you get: higher aggregate utilisation for bursty workloads, and no memory isolation at all. Consumers share the device’s memory, so one process can exhaust it and the others fail. There is no performance isolation either: a heavy neighbour makes your work slower, unpredictably.
Good for development environments, interactive notebooks, and anything where the consumers trust each other and nobody has a latency requirement. Bad for production inference with a service level objective.
Hardware partitioning
Some devices support being divided into instances with dedicated compute units, dedicated memory, and dedicated paths to that memory. Each instance appears as a separate device.
This is real isolation. A neighbour cannot take your memory or your bandwidth. Performance is predictable.
The costs: the partition layout is configured rather than dynamic, so you commit to a shape; partitions are fixed sizes rather than arbitrary fractions; and a partition is smaller than the whole device, so a job that would benefit from the full thing cannot have it while the partitioning is in place.
This is the right mechanism for multi-tenant inference and for giving several teams guaranteed capacity.
Full virtualization with a mediated driver
The hypervisor presents virtual devices to virtual machines, with the vendor’s software scheduling between them. Isolation at the virtual machine boundary, memory divided by profile, and live migration in some configurations.
This is the mechanism behind graphics virtualization for virtual desktops, and it is the one to use when the consumers are VMs rather than containers. It carries a licensing cost and a driver version matrix that must line up between host and guest.
Choosing
Ask two questions.
Do the consumers trust each other? If they are one team’s jobs, time-slicing is fine and cheapest. If they are different tenants, or if one is production, you need real isolation.
Is there a latency requirement? Anything with a service level objective needs partitioning or virtualization. Time-slicing gives you no guarantee and the violations will be intermittent, which is the worst kind.
What people get wrong
Sharing across training jobs. Distributed training synchronises; a job sharing devices with something else becomes the slow participant and holds up every other node. Training jobs should own their devices.
Assuming Kubernetes handles it. The default device plugin behaviour allocates whole devices. Sharing requires explicit configuration, and which mechanism you get depends on what you configured, not on what you assumed.
Partitioning too finely. Small partitions fit more tenants and each gets less memory. A partition that cannot hold the model is useless. Size partitions from the largest model that must run in them, not from the number of tenants you want.
Forgetting the memory is the constraint. Compute sharing is easy to reason about. Memory is the hard limit, and a device shared four ways has a quarter of the memory per consumer. For inference, that determines your context length and concurrency, which is the actual capacity of the service.