AI agent monitoring in production: traces, cheap judges, and the loop that compounds

I run a monitoring cockpit over a fleet of automations and agents: 36 read-only probes, a status lamp per system, a daily LLM-written health pulse, and a queue of recommendations waiting for a human decision. It cost weeks to get honest and about two dollars a month to run, and it has caught real failures. It has also lied to me at least three times, and the lies taught me more about monitoring agents than the catches did. This post is the field version of both lists.

• The core difference. A server is up or down. An agent can be up, fast, cheap, and confidently wrong. Uptime checks do not see the fourth state.
• The architecture that works. Traces for everything, human eyes for flagged runs only, cheap binary judges on a sample, and a loop that turns reviewed failures into regression tests.
• The part everyone skips. Recommendations without a decision stamp are theatre. I found 134 signals marked "reviewed" by an automatic timestamp, and zero of them had actually been read.
• The scariest finding. A control that always returns zero is worse than no control. I ran one for two months.
• The cost. The whole cockpit runs on small serverless pieces and one cheap model. Monitoring agents is an architecture problem, not a budget problem.

Every gate in my production readiness checklist assumes you can answer "what did the agent actually do?" after the fact. This post is about the machinery behind that answer.

A model upgrade can also change what a trace shows. The Sonnet 5.5 progress-update change is one example to check during migration.

If you use Codex for this work, the GPT-6.1 Sol and dots guide separates model choice, task handoff and plan limits.

Why normal monitoring misses agent failures

Classic observability watches four things: errors, latency, saturation, traffic. All four matter for agents too, and none of them catch the failure mode that actually hurts: the run that completed successfully, fast, within budget, and did the wrong thing. A summarizer that dropped the one paragraph that mattered. A classifier that has been quietly mislabeling one category for a week. A report agent that posted numbers from the wrong date range with total confidence.

Agents are nondeterministic, sensitive to small prompt and input changes, and fed by users who can type anything. The practical consequence: you cannot enumerate the failure states up front, which is the assumption classic alerting is built on. So the job changes from "alert when a known-bad state occurs" to "record enough about every run that an unknown-bad state can be found, explained, and turned into a test". That is a different architecture, not a bigger dashboard.

Traces are the source of truth, dashboards are the summary

The non-negotiable layer is the trace: for every run, what the agent saw, which tools it called with what arguments, what came back, and what it produced. Not a log line saying "run completed". The full trajectory. When something looks wrong three days later, the trace is the difference between an investigation and a shrug.

Attach the run costs and its accepted outcome to that trajectory to calculate AI agent cost per successful task. Failed attempts and review time belong in the same ledger.

Everything else in my setup is derived. The 36 probes are read-only checks over systems and their outputs, each one reduced to a lamp: green, yellow, red, or no-data. The lamp view answers "is anything on fire" in five seconds. The trace answers "what exactly happened". Confusing those two jobs is the first classic mistake: a wall of lamps with no trajectories underneath tells you that something broke and never tells you why.

One design rule that paid for itself: the probes mutate nothing. Read-only monitoring makes the risk conversation short, the review is about what a probe may read, never about what it could break, which is why the cockpit got built instead of postponed.

The review queue: human eyes on flagged runs only

Human review does not scale to every run, and it does not have to. The pattern that works is a narrow queue: runs get flagged into it by an alert, a failed probe, a user complaint, or a judge verdict, and a human reviews only the queue. Everything else stays in the trace store, searchable when needed.

For workflow failures entering that queue, n8n error handling covers the error signal, the alert path and the decision to reconcile an uncertain action before retrying it.

The queue needs one discipline to stay useful: every item leaves it with an explicit outcome, not by scrolling past. In my cockpit the recommendation queue records a decision per item: accepted, rejected, deferred, or implemented. That sounds bureaucratic until you see the alternative, which I did, and which gets its own section below.

LLM as a judge: cheap model, binary verdict, sampled traffic

Automated quality checks on agent output work, with three constraints I now treat as rules. Use a small, cheap model: the judge runs constantly and its job is narrow. Ask for a binary verdict: pass or fail against a written rubric, never a 1-to-5 score, because scale points drift and nobody can define the difference between a 3 and a 4 under pressure. And sample: judging every run of a high-volume agent buys little over judging a slice, at many times the cost.

Before a judge earns a place in the loop at all, it has to pass the same bar as any control: score it against a hand-labeled sample first, and only wire it in once it agrees with the humans often enough to be worth listening to. After that, the judge is still the component most likely to quietly rot. A rubric written in March judges March's failure modes; by August the agent fails differently. My working fix is unglamorous: when a human review disagrees with a judge verdict, that disagreement is itself a signal, and enough of them mean the rubric gets rewritten. The judge audits the agent, the queue audits the judge.

The daily pulse: a bird's eye view that admits uncertainty

Lamps and queues are reactive. The layer I use daily is a pulse: once a morning, a small model reads the health table, the last day of signals, and the calendar context, and writes a short brief: what changed, what needs a decision, what can wait. It is the "fly over the whole system" view, and it is the piece people ask about most when they see the setup, because it feels like a luxury and costs about a dollar a month.

Two design choices matter more than the model. First, the pulse has guards: when its own inputs are stale or a source is unreachable, it says "I am not sure" instead of narrating confidently over missing data, and a deterministic fallback produces a degraded brief if the model call fails outright. An unreliable summarizer of reliability data is a special kind of poison. Second, the pulse proposes, it never executes. Its recommendations land in the decision queue like everything else.

Reports need a delivery channel humans actually look at; mine land in a Telegram group, because background work is invisible until a message says what happened.

Recommendations need a decision stamp, or the loop is theatre

The single most useful audit I ran on my own cockpit: checking whether the review trail was real. It was not. 134 signals carried a "reviewed" timestamp, all stamped at the same time each morning by the pipeline itself. Zero had been read by a person. The field existed, the process it claimed to record did not. The fix was mechanical, the stamp now only gets written by an actual review action, but the lesson is bigger than the bug: any field that says a human was involved will drift toward lying unless writing it is coupled to the human act.

Same story on the recommendation side. The pulse generates suggestions; for weeks they accumulated with no decision recorded, which means the same suggestions kept coming back and nothing compounded. Once every recommendation required a stamp, the queue started shrinking and repetition became a signal in itself: the same issue showing up three times gets surfaced as one pattern instead of three separate lines. The loop from trace to fix only compounds if decisions are recorded where the machine can see them.

What "control" means in this setup

"Monitor and control" sounds like it should mean the system can act on an agent directly: pause it or roll it back. That is not what happens here, and the absence is a design choice. Nothing in this cockpit reaches the agent on its own. The probes mutate nothing, by design. The daily pulse proposes and never executes. Every recommendation goes through the queue, and an item only leaves it once a person records an outcome: accepted, rejected, deferred, or implemented. Control, in this architecture, is not a lever the monitoring layer pulls on its own. It is the requirement that nothing downstream happens without a human decision in between.

That is a narrower claim than "control" usually implies. A system that can pause or roll back an agent on its own reacts faster than one that waits for a person to clear a queue. What this setup buys instead is a recorded decision behind every recommendation that gets acted on, and a design that never lets the review layer become the thing making unsupervised changes. The next section uses the same word in a different sense: a control there is an audit check, something that is supposed to notice a bad state and say so. That is a different job from the human-decision requirement described here, and it is the audit kind that lied to me.

Two controls that lied to me

The control that always returned zero. One compliance check queried an audit trail for console-login events in my working region. It went green through two consecutive audits. The events were being recorded in a different region all along; the query could never have found anything, and when I finally asked the right region, there had been 38 real logins, every one of them justified. Nothing bad had happened, but if it had, the control would have shown zero anyway. The rule that came out of it: any check that returns "clean" must first prove it can detect a planted positive. A control that cannot fail is not a control, it is a sedative.

The probe that measured activity instead of process. A freshness probe on a mailbox pipeline went yellow every weekend. It measured "time since last email", so a quiet Saturday looked like a broken system. The pipeline was fine; the metric was answering a different question than the one on its label. And the dangerous half of the same defect points the other way: had the processor died while fresh mail kept arriving on its own, that metric would have stayed green. The old probe got demoted to informational, and a new one now watches the processor's own invocations. The general form: agents produce activity irregularly, so freshness of output is a bad proxy for health of process, and every probe label should name the thing it actually measures.

This is also where the diagram's third box lives, the part that makes the loop compound. Each reviewed failure left a permanent check behind: the fake stamps left a rule that a review field can only be written by a review action, the zero-control left a requirement that every clean-returning check first proves it can catch a planted positive, the weekend probe left an invocations probe with an honest label. None of those three failures can happen to me twice, not because I got wiser, but because the regression net got wider. That is the entire difference between monitoring that observes and monitoring that accumulates.

What this costs and where to start

The whole cockpit, probes, health table, daily pulse, alerting into one channel, runs on small serverless functions and one cheap model, around two dollars a month at my scale. The expensive part was never compute. It was the honesty work: calibrating probes that cried wolf, killing the fake review stamps, testing that controls can actually detect anything.

Starting order that would have saved me time:

1. Traces first. Full trajectories for every agent run, before any dashboard.
2. One alert channel. Everything pages into the one place a human reads.
3. Lamps over systems, read-only, each labeled with what it truly measures.
4. A review queue with mandatory outcomes. No stamp, no exit.
5. A sampled judge with a binary rubric, audited by the queue.
6. Every reviewed failure becomes a permanent check. This is the loop's compounding step; skip it and steps 1 through 5 just describe your outages in more detail.
7. A daily pulse last. It is the roof, not the foundation, and it is only as honest as the layers under it.

The verdict

Monitoring agents in production is mostly not an AI problem. It is an old discipline, observability, meeting a new failure mode, confident wrongness, and the winning moves are structural: record everything, review a narrow slice, automate the boring verdicts, and force every recommendation through a human decision that leaves a mark. The AI-specific parts, the judge and the pulse, are the cheapest components in my stack and the least trustworthy, which is why both run inside guardrails built from the boring parts. Watch the watchers: the two controls that lied to me were the two I never thought to check.