When a shared CI/CD product needs an upgrade, every team that consumed it has a stale copy. This automation found every deployed pipeline stack across the fleet and brought them all to the latest version — 500+ pipelines patched in parallel — safely, idempotently, and with a paper trail.
CI/CD infrastructure was distributed to teams as ServiceCatalog products. That's great for day one — and a versioning nightmare for day two. Each mandatory change (new security scanning stage, IMDSv2 enforcement, runtime upgrades) meant asking hundreds of teams to update their provisioned products. Most wouldn't. The fleet drifted.
discover plan execute verify ┌─────────────┐ ┌─────────────────┐ ┌───────────────┐ ┌──────────────┐ │ enumerate │───►│ diff provisioned│─►│ update product│─►│ pipeline runs│ │ provisioned │ │ version vs │ │ via Service- │ │ green? stack │ │ products in │ │ latest artifact │ │ Catalog API │ │ healthy? │ │ every acct │ │ skip if current │ │ per account │ │ → CSV report │ └─────────────┘ └─────────────────┘ └───────────────┘ └──────────────┘ ▲ STS assume-role per account · region-aware · resumable
The engine walks every account, finds provisioned CI/CD products, compares versions, and updates only what's stale. Every action lands in a per-account CSV — who was upgraded, who was skipped, who failed and why.
Idempotent by default. Re-running the tool is always safe: current stacks are skipped, failed ones retried. Partial runs don't leave the fleet in an unknown state.
ServiceCatalog as the control plane. Upgrades flow through the same governed API teams used to provision — permissions, constraints, and history all stay intact.
Batteries for the long tail. Companion scripts handle the odd cases: security-group rule cleanup, IMDS enforcement on stray EC2, token generation for multi-account API access, cross-account principal generation.
Human-shaped rollout. Account cohorts (by CSV) let risky changes soak in a small ring before fleet-wide execution.
A change that would have been hundreds of tickets and a quarter of chasing became a supervised run measured in hours. Compliance changes (scanning stages, IMDSv2) now reach 100% of the fleet, provably, with an audit CSV to show for it.