Not a chatbot with cluster access — an agent designed like an SRE works: plan the investigation, gather evidence in parallel, critique its own conclusion against live cluster state, and only answer when the evidence converges. Built by someone who has operated 150+ clusters and 500+ GPUs, for the questions those fleets generate all day.
Every platform team is one team supporting forty. The bottleneck isn't fixing things — it's the endless stream of questions: what's running here, why is that crashing, who's eating this node. Each one costs a senior engineer a context switch. k8sAgent's design goal is precise: answer the questions a mid-level SRE could answer, with the evidence a senior SRE would demand, at the cost of an API call.
That framing drives everything below. A demo agent answers questions; a production agent has to be right, provably, or say it isn't sure. The architecture is organized around that difference.
The heart of the system is a state machine — plan, act, observe, critique — that iterates until the answer is supported by evidence or the agent explicitly asks for clarification. No single-shot answers on diagnostic queries, ever:
Three properties make the loop production-shaped rather than demo-shaped:
Bounded. Hard iteration cap and a per-query token budget. An agent that can loop forever is an agent that can spend forever — the budget is enforced in the controller, not suggested in the prompt.
Self-critical. The critique step re-reads the assembled evidence against the original question: are all claims backed by a tool result? do any two results contradict? is a follow-up tool call cheaper than a wrong answer? Diagnosis-class queries route through this gate every iteration.
Honest under uncertainty. Below the confidence threshold, the agent asks a clarifying question or states exactly which evidence is missing. "I don't know yet, here's why" is a first-class terminal state — the one most agents are missing.
interaction layer web console · Slack · CLI (same API) │ intent router small model, temp 0 — domain (k8s/aws) + activity │ (list · describe · diagnose) — cheap, fails safe ▼ orchestrator LangGraph-style state machine — the convergence loop │ per-query budget · iteration cap · confidence gate ▼ MCP tool layer k8s_tool · aws_tool · file_tool — stdio contract, │ parallel fan-out, read-only ServiceAccount ▼ context engine YAML → schema extraction → LLM key-filter (batched, │ persistent cache) → projected JSON + _hint ▼ observability Langfuse traces · scores · prompt registry · evals
Model-agnostic by construction. LiteLLM abstracts the provider — Bedrock today, anything tomorrow — so model choice is a config value per pipeline stage: a small fast model for routing and schema filtering, a strong model for diagnosis. Each stage's model, prompt version, and cost are independently swappable and independently measured.
Context engineering as a subsystem. Raw kubectl -o yaml would drown any model. The context engine extracts each resource's schema, has a small LLM select relevant keys for this query (one batched call per iteration), projects the YAML down, and attaches an intent-aware _hint so formatting knowledge travels with the data. Schema decisions persist in a cache that survives restarts; trivial queries bypass the filter entirely via regex. Token spend per query drops by an order of magnitude before the answer model sees a byte.
Prompts change; without regression testing, every change is a gamble. The eval harness is built on Langfuse: a golden set of cluster questions with known-correct answers (seeded from real fleet scenarios — CrashLoopBackOff with bad image, OOMKilled with memory pressure, pending pods with taint mismatches), replayed against every prompt or model change. Each run scores correctness (does the answer match ground truth), groundedness (is every claim traceable to a tool result), and efficiency (iterations and tokens to convergence). A prompt change that improves tone but regresses groundedness doesn't ship. Every production trace feeds the same pipeline — real queries that converge slowly become tomorrow's eval cases. A gate run looks like this (representative output):
$ python -m evals.run --suite golden-v3 --gate groundedness>=0.98 suite: golden-v3 · 24 scenarios prompt: diagnose@v14 ───────────────────────────────────────────────────────────── correctness 23/24 (0.958) ✓ gate ≥ 0.95 groundedness 1.000 ✓ gate ≥ 0.98 efficiency avg 2.1 iterations · 9.3k tokens/query vs diagnose@v13: +1 correctness · −8% tokens RESULT: SHIP — promoted to prompt registry as diagnose@v14
Designed with the same posture as fleet tooling: the read path is scoped, the write path is gated.
The agent runs under a dedicated read-only RBAC ServiceAccount — its permissions are a Kubernetes object you can audit, not a promise in a prompt. Namespace scoping limits blast radius per deployment.
Mutations never execute directly — by design, the write path (roadmap v2) ships changes only as GitOps pull requests — the agent proposes the manifest diff, a human approves, ArgoCD applies. The agent gets the leverage of action with the audit trail and rollback story of GitOps, and an occasional wrong proposal costs a PR review instead of an incident.
Every interaction is traced end-to-end in Langfuse — who asked what, which tools ran, what the model saw, why it answered as it did. An agent you can't replay is an agent you can't trust in a postmortem.
The interface is designed around a rule: never show a conclusion without its evidence. Answer on the left; the tool results it cites and the live trace on the right — one click from any claim to the JSON that supports it:
The _hint generator reads the user's intent, not just the data — formatting knowledge travels with each tool result:
if format_type == "list": if any(w in q for w in ["how many", "count", "total"]): return f"User wants a count. One sentence: there are {item_count} items." if any(w in q for w in ["failing", "crashloop", "oomkilled"]): return "Present ALL items as a table: name, namespace, status, reason. " \ "Do not drop rows. Do not give fix steps — just the table." # "failing pods" = a table, NOT a diagnosis — activity wins over keywords