An EKS version upgrade is a multi-step, order-sensitive operation with real failure modes. Now do it for a fleet of 150+ clusters across many AWS accounts — every quarter. This orchestrator makes that a supervised batch job instead of a months-long manual campaign.
EKS versions expire. Miss the window and AWS force-upgrades you — on its schedule, not yours. A manual cluster upgrade (control plane → node groups → addons → validation) takes an engineer the better part of a day. Multiply by hundreds of clusters and the math simply doesn't work: the fleet would never be current.
A four-phase pipeline, where each phase runs concurrently across clusters and no phase starts until the previous one is verified:
phase 1 phase 2 phase 3 phase 4 ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ control │ ──► │ managed │ ──► │ addons │ ──► │ sanity check │ │ plane │ │ node │ │ (vpc-cni,│ │ nodes/pods/ │ │ upgrade │ │ groups │ │ coredns, │ │ CRDs/addons │ └──────────┘ └──────────┘ │ k-proxy) │ └──────────────┘ ▲ └──────────┘ │ │ SAML auth ► STS assume-role ▼ ┌─────────────┐ per target account ┌──────────────────┐ │ accounts.csv│ │ report.csv │ │ fleet targets│ │ pass / fail / why│ └─────────────┘ └──────────────────┘
The final phase is its own tool — a post-upgrade validation framework that checks node readiness, workload health, addon versions, and CRD integrity using the native Kubernetes Python client, so "upgrade done" means verified, not hoped.
Cross-account by design. SAML-federated auth mints STS credentials per target account — no long-lived keys, no manual role switching across accounts.
Thread-safe concurrency. Clusters upgrade in parallel with bounded worker pools and thread-safe result collection, so one slow cluster doesn't stall the fleet and one failed cluster doesn't corrupt the report.
Resumable & idempotent. Already-current clusters are detected and skipped; a re-run after a partial failure picks up exactly where it left off.
Companion tooling. Separate utilities handle addon-only updates and fleet-wide health snapshots, so routine maintenance doesn't require the full orchestrator.
Two details separate this from an upgrade script. First, the two-phase waiter: update_cluster_version is async and the cluster reports ACTIVE at the old version for a while after the call — so the waiter first waits for the cluster to leave ACTIVE, then for ACTIVE at the target version. Without that, a naive poller declares success before the upgrade begins.
Second, attach-don't-fail: a re-run that finds a cluster mid-upgrade from a previous partial run attaches to the in-flight update and waits, instead of erroring. Combined with per-phase timeouts, a --dry-run mode, and version comparison that skips current clusters, re-running the orchestrator is always safe:
# upgrade_control_plane.py — if mid-upgrade from a previous run, wait instead of failing if cluster_status == "UPDATING": wait_for_cluster_active(eks, cluster_name, target_version, cp_timeout) # helpers.py — semantic version compare decides skip-vs-upgrade if not version_lt(current_version, target_version): write_result({"cluster": name, "status": "SKIPPED", "reason": "already current"}) return
Fleet-wide upgrades became a scheduled, supervised batch operation with a CSV audit trail per run — instead of a quarter-long engineering campaign. Clusters stay inside the EKS support window, and validation is automatic rather than tribal knowledge.