Most organizations now run agents in production, and the consistent finding is that quality, not cost, is the barrier. For infrastructure agents the quality bar is brutal: a wrong answer about a cluster isn't a bad paragraph, it's a misdirected incident response. This is the architecture I use — built as k8sAgent — and the reasoning behind each layer.
The naive design — a strong model, cluster credentials, a system prompt saying "be careful" — fails in three characteristic ways. It hallucinates state, describing pods that don't exist because plausibility is the model's native register. It burns context, because raw kubectl -o yaml output is enormous and mostly irrelevant, so signal drowns. And it answers when it shouldn't, because nothing in a single forward pass distinguishes "I verified this" from "this sounds right."
Each failure names a required subsystem: grounding, context engineering, and a convergence loop with an explicit uncertainty exit. The literature converged on the same skeleton — ReAct-style reason/act cycles plus a reflection pass — but the pattern names undersell where the engineering actually lives. That's what this post is about.
query ─► route ─► plan ─► act (parallel tools) ─► observe ▲ │ │ ▼ └────── refine ◄─── critique gate │ converged ──────────┼─────────── low confidence ▼ ▼ answer + cited evidence ask / state what's missing
Three engineering decisions make this loop production-shaped rather than demo-shaped:
The critique gate is structural, not prompted. Reflection implemented as "please double-check your answer" in a prompt degrades under distribution shift. Implemented as a separate state with its own criteria — every claim mapped to a tool result, contradictions between results surfaced, marginal value of one more tool call estimated — it's testable in isolation and survives prompt drift. Diagnosis-class queries pass through it every iteration.
Budgets are enforced in the controller, not requested in the prompt. Iteration caps and per-query token budgets live in code. An agent whose loop can run away is an agent whose cost model is fiction — and "the prompt says stop after five tries" is not an enforcement mechanism.
Uncertainty is a terminal state. Below the confidence threshold the agent asks a clarifying question or states which evidence is missing. Designing "I don't know yet" as a first-class output — with the same UI weight as an answer — is the single cheapest reliability feature an agent can have, and the one most demos omit because it looks weak. It's the opposite: it's what lets people trust the answers that do come back.
An agent is a pipeline, and not every stage deserves the same model. Domain classification (Kubernetes vs AWS) and activity classification (list / describe / diagnose) run on a small, temperature-zero model — tens of tokens, milliseconds, fails safe to the conservative branch. The strong model is reserved for the reasoning stage where it earns its cost. This also unlocks per-activity system prompts: the presenter never drops rows and never editorializes; the inspector structures around status and errors; the diagnostician correlates evidence and normalizes units before recommending anything. Small prompts, separately testable — the opposite of the 3,000-token mega-prompt that handles everything badly.
The highest-leverage code in the system never calls the big model. Tool output goes through a pipeline: parse the YAML, extract top-level keys as a schema, have the small model select which keys matter for this query (batched — one call covers all tools per iteration), project the YAML down, and attach a _hint describing how this data should be presented. Schema decisions persist in a cache that survives restarts; trivial queries bypass the filter via regex. Token spend per query drops by an order of magnitude before the reasoning model sees a byte.
The _hint mechanism deserves specific attention because it solves a coupling problem: formatting knowledge travels with the data rather than accreting in the agent's prompt. Add a tool, and its output presents correctly without touching anything else. And the hints are intent-aware:
if format_type == "list": if any(w in q for w in ["how many", "count", "total"]): return f"User wants a count. One sentence: there are {item_count} items." if any(w in q for w in ["failing", "crashloop", "oomkilled"]): return "Present ALL items as a table: name, namespace, status, reason. " \ "Do not drop rows. Do not give fix steps — just the table." # "failing pods" = a table, NOT a diagnosis — activity wins over keywords
That comment is the hardest-won line in the codebase. Users say "failing" and mean show me, not fix it. Intent classification must respect the activity, not keyword-match its way into an unwanted diagnosis.
Prompt changes without regression testing are gambling with extra steps. The eval harness — built on Langfuse, which has emerged alongside LangGraph as the standard production pairing for traced, evaluated agents — replays a golden set of cluster scenarios (CrashLoopBackOff on a bad image, OOMKilled under memory pressure, Pending pods on taint mismatches) against every prompt or model change. Three scores per run: correctness against ground truth, groundedness (is every claim traceable to a tool result — the anti-hallucination metric), and efficiency (iterations and tokens to convergence). A change that improves tone but regresses groundedness doesn't ship. Production traces feed the same pipeline: queries that converge slowly become tomorrow's eval cases. This closed loop — trace, score, curate, gate — is the difference between prompt-tweaking and engineering.
Reads execute directly under a read-only RBAC ServiceAccount — the agent's authority is a Kubernetes object you can audit, not a promise in a prompt. This matters more than it looks: prompt injection via tool output is an open problem, and the sane mitigation for infrastructure agents is making the worst-case injection outcome a weird read, never a write.
Writes never execute at all. The write path emits GitOps pull requests — the agent proposes a manifest diff, a human approves, the reconciler applies. This slots the agent into the trust machinery that already exists (review, audit trail, rollback) instead of inventing a parallel approval system. A wrong proposal costs a code review; a wrong direct write costs a postmortem. Plan-then-execute with a human gate isn't a limitation of the design — it is the design.
Build the eval set before the second prompt. Put budgets in code on day one. Make "I don't know" a designed output. Spend your cleverness on context engineering, not prompt poetry — the model can't reason over what it can't see, and it reasons worst over everything at once. And decide the write-path security model before the demo works, because after the demo works, nobody wants to hear about it.