~ / work / eks-fleet-upgrades

Upgrading a 150+ cluster EKS fleet_

An EKS version upgrade is a multi-step, order-sensitive operation with real failure modes. Now do it for a fleet of 150+ clusters across many AWS accounts — every quarter. This orchestrator makes that a supervised batch job instead of a months-long manual campaign.

role
designer & author
scale
150+ EKS clusters · ~1,000 nodes · 30K+ pods · cross-account
stack
Python, boto3, Kubernetes client, SAML/STS cross-account auth, ThreadPoolExecutor

the problem

EKS versions expire. Miss the window and AWS force-upgrades you — on its schedule, not yours. A manual cluster upgrade (control plane → node groups → addons → validation) takes an engineer the better part of a day. Multiply by hundreds of clusters and the math simply doesn't work: the fleet would never be current.

the approach

A four-phase pipeline, where each phase runs concurrently across clusters and no phase starts until the previous one is verified:

 phase 1          phase 2          phase 3          phase 4
 ┌──────────┐     ┌──────────┐     ┌──────────┐     ┌──────────────┐
 │ control  │ ──► │ managed  │ ──► │ addons   │ ──► │ sanity check │
 │ plane    │     │ node     │     │ (vpc-cni,│     │ nodes/pods/  │
 │ upgrade  │     │ groups   │     │ coredns, │     │ CRDs/addons  │
 └──────────┘     └──────────┘     │ k-proxy) │     └──────────────┘
      ▲                            └──────────┘            │
      │            SAML auth ► STS assume-role            ▼
 ┌─────────────┐   per target account          ┌──────────────────┐
 │ accounts.csv│                                │ report.csv       │
 │ fleet targets│                                │ pass / fail / why│
 └─────────────┘                                └──────────────────┘

The final phase is its own tool — a post-upgrade validation framework that checks node readiness, workload health, addon versions, and CRD integrity using the native Kubernetes Python client, so "upgrade done" means verified, not hoped.

engineering details

Cross-account by design. SAML-federated auth mints STS credentials per target account — no long-lived keys, no manual role switching across accounts.

Thread-safe concurrency. Clusters upgrade in parallel with bounded worker pools and thread-safe result collection, so one slow cluster doesn't stall the fleet and one failed cluster doesn't corrupt the report.

Resumable & idempotent. Already-current clusters are detected and skipped; a re-run after a partial failure picks up exactly where it left off.

Companion tooling. Separate utilities handle addon-only updates and fleet-wide health snapshots, so routine maintenance doesn't require the full orchestrator.

from the code

Two details separate this from an upgrade script. First, the two-phase waiter: update_cluster_version is async and the cluster reports ACTIVE at the old version for a while after the call — so the waiter first waits for the cluster to leave ACTIVE, then for ACTIVE at the target version. Without that, a naive poller declares success before the upgrade begins.

Second, attach-don't-fail: a re-run that finds a cluster mid-upgrade from a previous partial run attaches to the in-flight update and waits, instead of erroring. Combined with per-phase timeouts, a --dry-run mode, and version comparison that skips current clusters, re-running the orchestrator is always safe:

python · the resumability contract
# upgrade_control_plane.py — if mid-upgrade from a previous run, wait instead of failing
if cluster_status == "UPDATING":
    wait_for_cluster_active(eks, cluster_name, target_version, cp_timeout)

# helpers.py — semantic version compare decides skip-vs-upgrade
if not version_lt(current_version, target_version):
    write_result({"cluster": name, "status": "SKIPPED", "reason": "already current"})
    return

impact

Fleet-wide upgrades became a scheduled, supervised batch operation with a CSV audit trail per run — instead of a quarter-long engineering campaign. Clusters stay inside the EKS support window, and validation is automatic rather than tribal knowledge.

PythonEKSboto3 SAML/STSThreadPoolExecutorIdempotency
← prev: ECS platform module next: GitOps addon platform →