Inventory-driven GitOps: the fleet as a directory tree_

the problem nobody models

Every Kubernetes cluster carries a platform layer — secrets operators, ingress controllers, observability agents, security scanners — before the first application lands. At fleet scale this layer has a shape most teams never write down: a sparse matrix of (cluster × addon × version). Cluster A runs Traefik v2 and the OTEL collector; cluster B runs Traefik v3 and adds a security agent; sixty clusters share a baseline; twelve have exceptions.

Hand-installed, that matrix lives nowhere — it's archaeology. Pipeline-installed, it lives in CI variables — it's archaeology with YAML. The architecture below makes the matrix a first-class, versioned artifact: a directory tree in Git.

three primitives, composed

Discovery: ApplicationSets with cluster generators. Clusters register with labels — environment, account, application code — and ApplicationSets enumerate them at reconcile time. Membership in the platform is a property of existing with the right labels, not of being listed in anyone's pipeline. This is the industry-standard answer to multi-cluster fan-out for good reason: the alternative, hand-created Applications per cluster, scales O(clusters × addons) in human attention.

Rendering: app-of-apps per environment. A Helm chart per environment (dev/qa/prod) renders one ArgoCD Application per addon per cluster. Addon versions promote dev → qa → prod exactly like application code, because they are code.

Truth: the per-cluster inventory. The decisive design choice. Chart logic lives in one repo; each cluster owns an enabler.yaml in an inventory repo declaring which addons it runs. ArgoCD's multi-source Applications staple the two together at render time:

sources:
  - path: helm-charts/dev                 # HOW addons deploy — chart repo
    helm:
      parameters:
        - { name: global.clusterName, value: "{{name}}" }
      valueFiles:                         # WHAT this cluster runs — inventory repo
        - $Values/inventory-dev/{{metadata.labels.account_alias}}/{{name}}/enabler.yaml
  - ref: Values

Read the templated path carefully — that's the whole architecture. The generator's label output selects a folder; the folder is the cluster's row of the matrix. Enabling an addon anywhere in the fleet is a one-line commit. Auditing the fleet is git log. Answering "which clusters run what" is git grep.

separating how from what

The two-repo split isn't tidiness — it's an access-control and change-velocity boundary. The chart repo changes rarely, reviewed by platform engineers, and a bug there has fleet-wide blast radius. The inventory repo changes constantly, one line at a time, with per-cluster blast radius — safe to delegate to cluster owners. Collapsing them into one repo forces both change classes through the same gate, and you end up either slowing trivial enables to platform-review speed or exposing chart logic to casual edits. The boundary in the repo layout is the org design.

scaling the control plane itself

At 150+ clusters, ArgoCD stops being an app you installed and becomes infrastructure you capacity-plan. The community numbers match our experience: shard the application controller at roughly one shard per 15–20 clusters (consistent-hashing shard assignment minimizes reshuffling as the fleet grows); watch the repo server — manifest generation is the silent bottleneck long before the controller is, since every addon × cluster combination is a render; and keep per-shard application counts bounded (community guidance clusters around a few hundred apps per shard). None of this is exotic — it's the same lesson as every control plane: the thing that deploys everything needs its own SLO, its own capacity model, and its own upgrade story. That last one bites hardest: ArgoCD upgrading ArgoCD is a chicken-and-egg you want rehearsed before 150 clusters depend on it.

failure modes, from experience

Inventory schema debt. Once 150 clusters depend on the shape of enabler.yaml, changing it is a migration with a compatibility window, not a refactor. Version the schema from day one; leave room for per-addon structured config, not just booleans.

Sync storms. A chart-repo change fans out to every cluster at once — which is the point, and also a thundering herd against the repo server and every cluster's API server. Progressive sync (environment waves) turns fan-out into a rollout.

The unmanaged escape hatch. The first time someone kubectl-patches an addon "just for now," self-heal reverts it and they learn the platform is real. Make the legitimate path (an inventory PR) faster than the illegitimate one, or people will fight the reconciler instead of using it.

references

→ Argo CD — high availability & scaling
→ Sharding clusters across Argo CD controller replicas
→ Argo CD — ApplicationSet controller