A reference architecture for running large-scale LLM batch inference on interruptible capacity — distilled from operating 500+ Spot GPUs at ~80% utilization and ~60% below on-demand cost.
GPU compute inverts the usual cloud cost model: with CPUs, engineering time dominates and infrastructure efficiency is a rounding error. With GPU fleets at inference scale, the infrastructure line is the P&L. Spot pricing offers up to ~90% discounts on that line — in exchange for a two-minute reclaim notice on any node, at any moment.
Most teams evaluate that trade as a reliability question and reject it. That's a category error. The correct framing is architectural: can you build a system in which interruption is a scheduled event rather than a failure? If yes, the discount is structural and permanent. If no, no amount of Spot "best effort" will save you — you'll pay the discount back in re-run compute and engineer attention.
What follows is the four-layer architecture that makes the answer yes.
Every (instance type × availability zone) pair is an independent Spot capacity pool with its own supply curve and interruption behavior. A NodePool restricted to one GPU type in one AZ is a bet on a single volatile market. Diversification isn't a nice-to-have; Karpenter's own Spot-to-Spot consolidation refuses to operate without meaningful instance-type diversity (the documented guidance is 15+ types) precisely because narrow pools concentrate interruption risk on whatever is scarcest.
kind: NodePool spec: template: spec: requirements: - { key: karpenter.sh/capacity-type, operator: In, values: [spot] } - { key: node.kubernetes.io/instance-type, operator: In, values: [g5.xlarge … p4d.24xlarge] } # wide: many pools - { key: topology.kubernetes.io/zone, operator: In, values: [us-east-1a, 1b, 1c] } # every AZ = more pools taints: - { key: nvidia.com/gpu, effect: NoSchedule } # GPU metal is for GPU work
The subtlety most write-ups miss: consolidation policy on GPUs is a different problem than on CPUs. Karpenter's utilization-driven consolidation assumes pod resource usage is a meaningful signal; on GPU nodes it mostly isn't — a GPU can be 100% busy while the pod's CPU/memory look idle. For inference fleets the working configuration is empty-node consolidation with a deliberate consolidateAfter delay, plus karpenter.sh/do-not-disrupt on batch pods that must not be voluntarily moved mid-run. You want Karpenter aggressive about empty nodes and conservative about working ones.
At 500-GPU scale, the scheduler's job is fighting two entropies. Fragmentation: an 8-GPU node running a 1-GPU pod has 7 GPUs you're paying for and can't sell to an 8-GPU job — bin-packing keeps multi-GPU capacity coherent. Correlation: if a job class lands entirely in one Spot pool, one reclaim wave kills the whole class — topology spread constraints across pools turn a correlated failure into a partial one.
These pull in opposite directions (packing concentrates; spreading dilutes), which is why they must be policy, not defaults: pack within a node, spread across pools. Getting this wrong doesn't page anyone — it just quietly costs 15–20 points of utilization, which at GPU prices is a headcount.
Infrastructure alone can't make interruption free; the workload has to hold up its end. The contract has three clauses, and they're checkable in review:
Progress is durable. Batch inference checkpoints its cursor externally — finished shards are never recomputed. The unit of loss on interruption is one in-flight shard, not one job.
Shutdown is graceful. The two-minute reclaim notice is an event you subscribe to, not trivia: drain, flush the current shard, requeue the remainder. Kubernetes gives you the machinery (termination grace, preStop); the platform's job is making the default path correct so forty teams don't reimplement it badly.
Retry is idempotent. Re-running a shard must be safe. If a job can't tolerate its own retry, it can't run on Spot — and honestly, it's one bad deploy away from incident on on-demand too.
The compounding effect is what matters: once retries are cheap, interruption stops being a reliability event at all. It becomes a scheduling event — and layer 1 already handles scheduling.
The metric that governs the platform isn't availability — batch inference doesn't have users refreshing a page — it's sustained GPU utilization. DCGM exporter feeds per-GPU utilization into Prometheus; the platform's SLO is a utilization floor, alerted and defended like uptime elsewhere. This closes the loop with every layer above: utilization dropping means fragmentation (layer 2), over-provisioned acquisition (layer 1), or workloads idling on checkpoint I/O (layer 3). One number, and its failure modes point at their owning layer.
Result, held over time rather than in a benchmark week: ~80% sustained utilization across 500+ GPUs at ~60% below the on-demand bill.
The "reliable Spot" fallacy — hand-picking the "least interrupted" instance type. Interruption rates are non-stationary; yesterday's calm pool is tomorrow's squeeze. Diversity beats prediction.
Checkpointing to local disk — durable progress on an ephemeral node is not durable progress.
Disabling consolidation entirely — panic response to one bad disruption. You keep the fragmentation tax forever to avoid a problem that do-not-disrupt annotations solve surgically.
Utilization dashboards without an SLO — a graph nobody defends decays into wallpaper. If 80% is the target, page below it.