k8lens: an in-cluster health radar_

the problem

A pod can be Running while its HTTP endpoint returns 503s. Kubernetes-level health (probes, conditions, replica counts) and user-level health (does the URL actually respond?) are different questions — and most monitoring setups only answer the first. At 150-cluster scale, "the dashboard was green but the service was down" stops being an anecdote and becomes a pattern.

the design

 K8sHealthMonitor (in-cluster pod, Helm-deployed)
      │
      ├── discovers ingresses + services via the K8s API
      │     └── skip-annotations & whitelists filter noise
      │
      ├── async fan-out — aiohttp session, asyncio.gather
      │     └── hundreds of endpoints probed in seconds
      │
      ├── repeats every check_interval (default 60s)
      │
      ▼
 Prometheus metrics ──► alerting · dashboards · history

Discovery, not configuration. k8lens reads the cluster's own ingress and service objects — new services are monitored the moment they exist, with annotations to opt out rather than config to opt in.

Async where it counts. One aiohttp session, bounded concurrent probes via asyncio.gather with exception isolation — a hanging endpoint can't stall the sweep, and a full-cluster pass completes in seconds rather than the minutes a sequential prober would need.

Metrics as the interface. Results export as Prometheus metrics, so "is it up?" plugs into the same alerting, dashboards, and history as every other signal — no bespoke UI to maintain.

engineering details

Kubernetes-native filtering. Namespace scoping, whitelists, and skip-annotations on resources mean teams control their own monitoring exposure with the primitives they already use.

Packaged like a platform component. Dockerfile + Helm chart + RBAC manifests — installable on any cluster with one helm install, and delivered fleet-wide through the GitOps addon platform: enabling k8lens on a cluster is one line in that cluster's enabler.yaml.

Metric cardinality is managed, not accidental. Counters are labeled by type, status, and namespace — deliberately not by endpoint URL or pod name, so a cluster with thousands of services doesn't detonate the Prometheus TSDB. The same discipline that trimmed metric ingestion costs across the 150-cluster observability rollout applies here.

operating profile

sweep interval

configurable, 60s default — full ingress + service pass per cycle

sweep latency

seconds for hundreds of endpoints (bounded async fan-out, shared session)

failure isolation

per-probe exception capture — a hanging endpoint is a data point, never a crashed cycle

opt-out surface

resource annotations + whitelist — teams control exposure with K8s primitives

tls posture

verification configurable per target class; relaxed only for self-signed internal infra

delivery

Helm chart · RBAC-scoped ServiceAccount · rolled out via GitOps inventory

from the code

async def _check_single_service(self, session, service, host, port):
    key = f"{service.metadata.namespace}/{service.metadata.name}"
    for path in SERVICE_HEALTH_PATHS:        # /health, /healthz, /ping…
        try:
            async with session.get(f"http://{host}:{port}{path}") as resp:
                if resp.status == 200:
                    health_check_counter.labels(type="service",
                        status="success", namespace=namespace).inc()
                    return key, True, namespace, name
        except Exception:
            continue                       # failure is a data point, not a crash

Probes address services by in-cluster DNS (name.namespace.svc.cluster.local) — measuring the same network path other workloads use, not the view from outside the ingress. The sweep wraps these in asyncio.gather(..., return_exceptions=True) so one hanging endpoint becomes an unhealthy-row in the results instead of killing the cycle.

impact

k8lens closes the gap between "Kubernetes says healthy" and "users say down." As part of the wider fleet observability suite, it's one of the tools behind an uptime platform covering 2,500+ endpoints — with the answer to "is it actually up?" arriving as a metric, not a support ticket.