Upgrading one cluster is a task. Upgrading 150+ is a control problem — and it deserves a doctrine, not a longer runbook. This is the one I built and operated at fleet scale.
The forcing function is contractual, not technical. EKS gives each Kubernetes minor version a 26-month lifecycle — 14 months of standard support, then 12 of paid extended support at a premium that compounds per cluster. Fall off the end and AWS force-upgrades your control plane on its schedule, not yours. Kubernetes itself ships three minors a year, and upgrades are strictly sequential: 1.30 → 1.33 means three complete upgrade campaigns, no skipping.
Run the arithmetic at fleet scale: 150 clusters × one careful engineer-day per manual upgrade × three minors a year is more upgrade-days than a platform team has days. Hire more people and the fleet grows faster than the team. The only stable solution changes the unit of work: not "an engineer upgrades a cluster" but "an orchestrator upgrades the fleet; engineers supervise rings."
pre-flight upgrade insights · deprecated-API scan · addon │ compatibility matrix · dry-run across the fleet ▼ ring 0 sandbox clusters — burn-in, days ▼ ring 1 dev/qa — wide coverage, real workloads, low blast ▼ ring 2 prod, cohorted — bounded concurrency per cohort │ │ per cluster, strictly ordered phases: │ control plane → node groups → addons → verification ▼ artifact machine-readable result per cluster per phase — the fleet's upgrade state is a queryable dataset
Two axes matter and they're orthogonal. Rings bound correlated risk — a bad interaction between a Kubernetes minor and your CNI version should be discovered on ring 0's five clusters, not ring 2's hundred. Phases encode the dependency order that Kubernetes itself imposes: control plane before kubelets (version skew policy), kubelets before addon versions that assume them, and verification before "done" gets said out loud.
Most upgrade damage is discoverable in advance. The pre-flight stage aggregates, per cluster: EKS upgrade insights (the control plane's own report of deprecated API usage against the target version), the addon compatibility matrix for vpc-cni / CoreDNS / kube-proxy, and a fleet-wide --dry-run that renders exactly what would change. The output is a go/no-go list — clusters with blocking findings get tickets before the campaign, not incidents during it. Fleet tooling without a dry run isn't automation; it's a loaded weapon with a nicer grip.
1. Idempotency — the run you can always re-run. Version comparison decides skip-vs-act per cluster; a partial failure resumes exactly where it stopped. The deeper case: a re-run may find a cluster mid-upgrade from the previous attempt. The orchestrator attaches to the in-flight update and waits, rather than failing:
# if mid-upgrade from a previous run — attach, don't fail if cluster_status == "UPDATING": wait_for_cluster_active(eks, cluster_name, target_version, cp_timeout) # semantic version compare decides skip-vs-upgrade if not version_lt(current_version, target_version): write_result({"cluster": name, "status": "SKIPPED", "reason": "already current"}) return
2. Correct waiting — async APIs lie briefly. update_cluster_version returns immediately, and for a window afterward the cluster still reports ACTIVE at the old version. A naive "wait for ACTIVE" poller declares victory before the upgrade begins. The fix is a two-phase waiter — first wait to leave ACTIVE, then wait to return at the target version:
seen_non_active = False while time.time() - start < timeout: status, ver = describe(cluster) if status != "ACTIVE": seen_non_active = True # phase 1: it actually started if status == "ACTIVE" and seen_non_active and ver == target_version: return True # phase 2: done, provably if status == "FAILED": raise RuntimeError(...)
This is a general lesson about cloud control planes, not an EKS quirk: every async mutation needs a waiter that distinguishes "hasn't started" from "finished."
3. Bounded concurrency — parallel, with a queue. A ThreadPoolExecutor caps in-flight upgrades per cohort; as_completed harvests results as they land. One slow cluster doesn't stall the ring; one failed cluster becomes a row in the report, not a dead campaign. The cap is a risk dial you turn per ring — wide for dev, narrow for prod.
"The API said the upgrade succeeded" and "the cluster works" are different propositions, and the gap between them is where fleet campaigns quietly rot. The final phase is a post-upgrade validation framework — native Kubernetes client, not kubectl parsing — asserting node readiness, workload health, addon versions, and CRD integrity, pass/fail per check, machine-readable. An upgrade is done when validation proves the cluster still does its job. Nothing else counts.
The artifact this produces is the doctrine's quiet superpower: the fleet's upgrade state becomes a dataset. "Are we exposed to the extended-support fee next quarter?" is a query, not a meeting.
Nothing above is Kubernetes-specific. Pre-flight, rings, ordered phases, idempotent execution, correct waiting, bounded concurrency, verification, artifacts — that eight-step shape turns any scary fleet mutation into a supervised batch job. The same skeleton ran our 500-pipeline CI/CD modernization and a fleet-wide security-agent rollout. Build it once, and every subsequent campaign is configuration.