~ / blog / gpu-optimization-stack gpu infrastructure · study · 8 min read

The 4-layer GPU optimization stack_

Most organizations waste 70–90% of the GPU capacity they pay for — because they treat GPUs like monolithic, indivisible blocks. This is my breakdown of the winning stack for GPU orchestration on Kubernetes: four layers that turn static GPUs into an elastic compute fabric.

discuss on LinkedIn ↗

the waste nobody budgets for

We solved this exact problem for CPUs two decades ago — virtualization turned rigid servers into fluid capacity, and nobody has thought about "which physical box" since. GPUs are at the same turning point today, except the stakes are an order of magnitude higher: the wasted unit isn't a $2,000 server, it's a five-figure accelerator sitting 80% idle while another team's training job waits in a queue.

The failure isn't any single tool — it's treating utilization as one problem. It's four, stacked: how you slice hardware, how you queue work, how you provision nodes, and how you see what's actually happening. Each layer has a winning answer on Kubernetes today.

Layer 1 — Hardware Slicing MIG + DRA · time-slicing Dynamically partition GPUs into smaller, isolated instances — right when they're needed. Layer 2 — Smart Queuing Kueue · cohorts · gang scheduling Batch AI workloads so jobs start whole — no deadlocks, no idle GPUs held hostage by pending pods. Layer 3 — Provisioning Karpenter + KEDA · groupless · event-driven Launch and bin-pack the most cost-effective GPU nodes from real demand — and only then. Layer 4 — Visibility & Hygiene DCGM + GPU reaper Monitor real GPU health and power draw; hunt down "zombie" pods squatting on expensive silicon. static GPUs ──► elastic compute fabric

layer 1 — hardware slicing: MIG + DRA

The default Kubernetes contract — nvidia.com/gpu: 1 — hands a whole accelerator to a pod that may need a fifth of it. Two mechanisms break that rigidity. Time-slicing is soft orchestration: multiple pods share a GPU with no isolation — fine for bursty dev workloads, dangerous for anything latency-sensitive. NVIDIA MIG (Multi-Instance GPU) is the real thing: hardware-level partitioning of a single GPU into isolated instances with dedicated memory and compute slices.

The piece that makes MIG operationally sane is Dynamic Resource Allocation (DRA) — Kubernetes' newer resource model that treats devices as first-class, parameterized, claimable resources instead of opaque integer counters. MIG + DRA means partitions are created when workloads claim them, not pre-carved by an admin guessing the demand mix in advance. That's the CPU-virtualization moment for GPUs: the hardware stops dictating the workload shape.

layer 2 — smart queuing: Kueue

The default scheduler places pods; AI workloads are jobs. That mismatch produces the classic distributed-training deadlock — and it's worth seeing concretely:

the deadlock, concretely
Team A requests 8 GPUs for training  →  only 6 free  →  job PENDING
Team B holds 4 GPUs                  →  idle, allocated, untouchable

Result: 10 GPUs in the cluster doing nothing, both teams blocked.

Kueue fixes both halves. Gang admission means a job is admitted only when its entire GPU quota can be satisfied — no partial starts squatting on capacity they can't use. Cohorts mean Team B's idle quota is borrowable: ClusterQueues in a cohort lend unused resources to each other and reclaim them when the owner needs them back. Utilization stops being a per-team property and becomes a fleet property — which is the entire point of running a shared platform.

layer 3 — provisioning: Karpenter + KEDA

Traditional node-group autoscalers make you pre-declare the shapes of capacity you might want — which, for GPU fleets with heterogeneous workloads, means either sprawl or starvation. Karpenter's groupless provisioning inverts it: look at the pending pods, compute the cheapest viable node that satisfies them, launch exactly that, bin-pack tightly. For the Spot-heavy version of this layer — diversity math, interruption handling, consolidation tuned for GPUs — see the companion deep-dive, The Interruption-Native GPU Platform.

KEDA completes the loop for event-driven inference: scale pods on external signals — queue length, Kafka lag, request depth — and let Karpenter chase the pod demand with nodes. The chain is demand → pods → nodes, each link reactive. Nothing is provisioned on a guess; everything is provisioned on a signal.

layer 4 — visibility & hygiene: DCGM + the reaper

You can't fix what you can't see — and GPU waste is uniquely invisible, because a "Running" pod holding a GPU looks identical to a working one from the orchestrator's view. NVIDIA DCGM gives you the ground truth: per-GPU utilization, memory pressure, power draw, thermals — exported to Prometheus, where a GPU drawing idle wattage while allocated becomes a visible, alertable fact.

Then act on it: a GPU reaper — a policy job that hunts pods holding accelerators with sustained near-zero utilization ("zombies": crashed notebooks, finished experiments nobody cleaned up, hung training loops) and evicts them by policy. It's the unglamorous layer, and in my experience the fastest payback: zombie hunting is free capacity you already bought.

the bottom line

Each layer compounds the others: slicing makes the units fit the work, queuing keeps the units busy, provisioning keeps the units cheap, and visibility keeps everyone honest. The destination is a mental-model shift — stop viewing clusters as rigid collections of servers and start treating them as a fluid, elastic fabric of compute. That shift is what separates experimental ML infrastructure from production-grade AI platforms; it's the same philosophy that ran 500+ Spot GPUs at ~80% utilization in the inference platform case study.

references

NVIDIA — MIG user guide
Kubernetes — Dynamic Resource Allocation
Kueue — batch admission, quotas & cohorts
Karpenter · KEDA
NVIDIA DCGM
the original LinkedIn post — join the discussion

KubernetesGPU optimizationMIG · DRA KueueKarpenter · KEDAFinOps
← prev: kro talk write-up next: interruption-native GPUs →