An LLM inference platform on 500+ Spot GPUs_

the problem

LLM inference at batch scale is a scheduling problem wearing a GPU costume. On-demand GPU capacity is ruinously expensive; Spot capacity is ~70% cheaper but can be reclaimed with two minutes' notice. Naïve setups either burn money on idle on-demand GPUs or watch long-running batch jobs die to interruptions and start over.

the approach

 batch queue ──► K8s scheduler + custom policies
                    │  bin-packing, GPU-aware placement,
                    │  topology spread across Spot pools
                    ▼
 Karpenter ──► provisions cheapest viable Spot GPU capacity
                    │  diversified instance types & AZs
                    ▼
 fault-tolerance layer
   ├── checkpointing & retry strategies for interrupted work
   ├── graceful drain on 2-min Spot reclaim notice
   └── requeue without losing progress
                    ▼
 DCGM ► Prometheus ► Grafana — utilization is a first-class SLO

The core insight: treat interruption as a normal event, not a failure. Once retry and checkpointing are cheap, Spot's discount is nearly free money — and utilization, not availability, becomes the metric to optimize.

engineering details

Scheduling policies tuned for GPUs. Bin-packing keeps fragmentation down so 500+ GPUs act like a coherent pool instead of stranded singles; spread constraints keep any one Spot pool reclaim from taking out a whole job class.

Karpenter for elasticity. Node capacity follows the queue — diversified across instance types and AZs so no single Spot market squeeze stalls the platform.

Utilization as an SLO. DCGM metrics flow into Prometheus/Grafana; ~80% sustained utilization is monitored and defended like uptime would be elsewhere.

Batch-first fault tolerance. Retry strategies and interruption handling mean large inference batches complete despite churn underneath them.

from the platform config

Two artifacts carry most of the platform's behavior — the Karpenter NodePool that buys the Spot market wide, and the job template that makes interruption survivable (representative shapes, values simplified):

kind: NodePool
spec:
  template:
    spec:
      requirements:
        - { key: karpenter.sh/capacity-type, operator: In, values: [spot] }
        - { key: node.kubernetes.io/instance-type, operator: In,
            values: [g5.xlarge … p4d.24xlarge] }    # wide list = many Spot pools
        - { key: topology.kubernetes.io/zone, operator: In,
            values: [us-east-1a, 1b, 1c] }
      taints:
        - { key: nvidia.com/gpu, effect: NoSchedule }
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized    # fight fragmentation

kind: Job
spec:
  backoffLimit: 6                       # retries are the plan, not the exception
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"   # no voluntary moves mid-batch
    spec:
      terminationGracePeriodSeconds: 110    # inside the 2-min Spot notice
      containers:
        - env:
            - { name: CHECKPOINT_URI, value: s3://…/cursor }  # durable progress
          lifecycle:
            preStop: { exec: { command: ["flush-and-requeue"] } }

impact

~60% compute cost reduction at ~80% sustained GPU utilization — on the most expensive resource class in the cloud, at 500+ GPU scale, while AI teams ship without thinking about capacity at all.

Why ~60% when Spot discounts run 70–90%? Deliberately: the delta pays for checkpointing overhead, requeue time after reclaims, and diversified instance selection that sometimes picks a pricier pool for availability. The discount you keep is the one that survives real workloads — a headline number that ignores those costs is a benchmark, not a platform.