Orchestrator state and user experience are different facts, and the gap between them is where the worst incidents hide. A design framework for in-cluster synthetic probing — and the decisions behind k8lens, one of the tools feeding a 2,500+ endpoint uptime platform.
Kubernetes gives you three verdicts about a workload — the pod phase, the probe results, the endpoint membership — and all three can be green while users get 503s. Liveness probes check what the container author thought to check; readiness gates traffic but not correctness; Running means the process exists. None of them answer the question an SRE actually gets paged for: does an HTTP request to this service, over the real network path, return a healthy response right now?
That question has exactly one honest measurement: send the request and look. Everything else is inference from proxies. Synthetic probing is unfashionable next to traces and eBPF — and it remains the only signal that measures the thing users experience, which is why the mature stack treats it as a layer, not an alternative: metrics say how it's failing, traces say where, probes say whether. You want "whether" from the most boring, most trustworthy component you own.
External probing (the Pingdoms of the world, or blackbox exporter outside the cluster) measures the full path — DNS, CDN, ingress — which is right for the public edge and wrong for diagnosis: an external failure tells you something in six layers broke. In-cluster probing measures the path your services actually use to talk to each other — cluster DNS (name.namespace.svc.cluster.local), kube-proxy, the pod network. The two disagree exactly when the diagnosis is interesting: external-red/internal-green is an edge problem; external-green/internal-red is east-west rot that external monitoring will never see. Run both; alert on the delta.
k8lens is the in-cluster half, and its defining choice is discovery over configuration: it enumerates the cluster's own Ingress and Service objects at sweep time and probes what it finds. Target lists are the config-drift failure mode of every probing setup I've seen — the service that pages you is always the one nobody registered. The cluster already knows what's running; ask it. Opt-out is a resource annotation, so teams manage their exposure with Kubernetes primitives rather than tickets against a config file.
A sequential sweep with polite timeouts takes tens of minutes at hundreds of endpoints — useless against a 60-second interval. The async fan-out (one shared aiohttp session, bounded asyncio.gather) is table stakes; the two decisions that matter are subtler:
async def _check_single_service(self, session, service, host, port): key = f"{service.metadata.namespace}/{service.metadata.name}" for path in SERVICE_HEALTH_PATHS: # /health, /healthz, /ping… try: async with session.get(f"http://{host}:{port}{path}") as resp: if resp.status == 200: health_check_counter.labels(type="service", status="success", namespace=namespace).inc() return key, True, namespace, name except Exception: continue # failure is a data point, not a crash
Exception isolation as policy. gather(..., return_exceptions=True) plus per-probe capture means a hanging endpoint becomes an unhealthy-row in the results. The prober is the component whose own failure model must be strictly stronger than the failures it measures — a monitor that dies on the outage it was built to detect has negative value, because it taught people to rely on it.
One session, many requests. Connection pooling keeps hundreds of concurrent probes from exhausting sockets or hammering cluster DNS. The line between "efficient monitor" and "accidental load test" is a session object.
Every label combination on a Prometheus metric is a new time series, stored and indexed forever-ish. Labeling probe results by endpoint URL or pod name at cluster scale detonates the TSDB — the classic self-inflicted observability outage. k8lens labels by type, status, namespace — enough to alert and aggregate, cheap enough to run on 150 clusters. Per-endpoint detail lives in logs, where high cardinality is priced honestly. The same discipline (aggressive ingestion trimming, cardinality budgets) is what kept the fleet-wide Managed Prometheus rollout affordable; a metric schema is a cost model whether you designed one or not.
And metrics are the product, not a byproduct: k8lens ships no UI. Results land in Prometheus, where they get the alerting, dashboards, retention, and SLO math every other signal gets — probe success ratios feed availability SLOs and multi-window burn-rate alerts for free. Building a bespoke monitoring UI is how side tools die; exporting metrics is how they get adopted.
Four principles, portable to any probing system: measure the request, not the proxy (synthetic HTTP over inferred state); discover, don't configure (the platform's own API is the target list); fail as data (the prober outlives what it probes); emit into the paved metric path (no bespoke UI, cardinality budgeted). Everything else — intervals, paths, TLS posture — is configuration.