Most organizations waste 70–90% of the GPU capacity they pay for — because they treat GPUs like monolithic, indivisible blocks. This is my breakdown of the winning stack for GPU orchestration on Kubernetes: four layers that turn static GPUs into an elastic compute fabric.
We solved this exact problem for CPUs two decades ago — virtualization turned rigid servers into fluid capacity, and nobody has thought about "which physical box" since. GPUs are at the same turning point today, except the stakes are an order of magnitude higher: the wasted unit isn't a $2,000 server, it's a five-figure accelerator sitting 80% idle while another team's training job waits in a queue.
The failure isn't any single tool — it's treating utilization as one problem. It's four, stacked: how you slice hardware, how you queue work, how you provision nodes, and how you see what's actually happening. Each layer has a winning answer on Kubernetes today.
The default Kubernetes contract — nvidia.com/gpu: 1 — hands a whole accelerator to a pod that may need a fifth of it. Two mechanisms break that rigidity. Time-slicing is soft orchestration: multiple pods share a GPU with no isolation — fine for bursty dev workloads, dangerous for anything latency-sensitive. NVIDIA MIG (Multi-Instance GPU) is the real thing: hardware-level partitioning of a single GPU into isolated instances with dedicated memory and compute slices.
The piece that makes MIG operationally sane is Dynamic Resource Allocation (DRA) — Kubernetes' newer resource model that treats devices as first-class, parameterized, claimable resources instead of opaque integer counters. MIG + DRA means partitions are created when workloads claim them, not pre-carved by an admin guessing the demand mix in advance. That's the CPU-virtualization moment for GPUs: the hardware stops dictating the workload shape.
The default scheduler places pods; AI workloads are jobs. That mismatch produces the classic distributed-training deadlock — and it's worth seeing concretely:
Team A requests 8 GPUs for training → only 6 free → job PENDING Team B holds 4 GPUs → idle, allocated, untouchable Result: 10 GPUs in the cluster doing nothing, both teams blocked.
Kueue fixes both halves. Gang admission means a job is admitted only when its entire GPU quota can be satisfied — no partial starts squatting on capacity they can't use. Cohorts mean Team B's idle quota is borrowable: ClusterQueues in a cohort lend unused resources to each other and reclaim them when the owner needs them back. Utilization stops being a per-team property and becomes a fleet property — which is the entire point of running a shared platform.
Traditional node-group autoscalers make you pre-declare the shapes of capacity you might want — which, for GPU fleets with heterogeneous workloads, means either sprawl or starvation. Karpenter's groupless provisioning inverts it: look at the pending pods, compute the cheapest viable node that satisfies them, launch exactly that, bin-pack tightly. For the Spot-heavy version of this layer — diversity math, interruption handling, consolidation tuned for GPUs — see the companion deep-dive, The Interruption-Native GPU Platform.
KEDA completes the loop for event-driven inference: scale pods on external signals — queue length, Kafka lag, request depth — and let Karpenter chase the pod demand with nodes. The chain is demand → pods → nodes, each link reactive. Nothing is provisioned on a guess; everything is provisioned on a signal.
You can't fix what you can't see — and GPU waste is uniquely invisible, because a "Running" pod holding a GPU looks identical to a working one from the orchestrator's view. NVIDIA DCGM gives you the ground truth: per-GPU utilization, memory pressure, power draw, thermals — exported to Prometheus, where a GPU drawing idle wattage while allocated becomes a visible, alertable fact.
Then act on it: a GPU reaper — a policy job that hunts pods holding accelerators with sustained near-zero utilization ("zombies": crashed notebooks, finished experiments nobody cleaned up, hung training loops) and evicts them by policy. It's the unglamorous layer, and in my experience the fastest payback: zombie hunting is free capacity you already bought.
Each layer compounds the others: slicing makes the units fit the work, queuing keeps the units busy, provisioning keeps the units cheap, and visibility keeps everyone honest. The destination is a mental-model shift — stop viewing clusters as rigid collections of servers and start treating them as a fluid, elastic fabric of compute. That shift is what separates experimental ML infrastructure from production-grade AI platforms; it's the same philosophy that ran 500+ Spot GPUs at ~80% utilization in the inference platform case study.