Agent evaluation that catches bad work before it ships

Most agent reviews start at the end: did the answer look right, did the code compile, did the ticket close, did the document sound plausible. That misses the part of agent behavior that usually predicts future failures. The trace often shows whether the agent inspected the right evidence, used tools for a reason, changed direction after a failure, and stopped when the work was no longer improving.

A system can produce a passable final answer while still behaving badly. It may search the same source six times, ignore an error, claim completion without checking the artifact, or widen scope until the original task disappears. Those are not style issues. They are product reliability issues because they create cost, latency, user confusion, and avoidable operational risk.

Low-quality agent work has recognizable shapes. One common shape is the unsupported completion claim: the agent says the task is done, but the trace contains no test run, file check, source citation, or domain-specific confirmation. Another is the repeated action loop: the agent keeps issuing similar searches, retries, or edits while the information state remains unchanged.

A third shape is weak recovery. Failures are normal in agent workflows, especially when tools, APIs, files, permissions, or search results are involved. The problem is not that a call fails. The problem is when the next action is the same action with a different phrasing, or when the agent hides the failure by moving to a generic answer.

A useful agent evaluation should combine deterministic trace checks with calibrated review. Deterministic checks catch the obvious and repeatable issues: required observations, forbidden tools, turn budgets, retry budgets, verification events, and stop conditions. Human or LLM review can then focus on higher-judgment questions such as whether the recovery strategy was reasonable or whether the final synthesis reflected the evidence.

The point is not to create a single magic score. The point is to make failure legible. A release review should be able to say: this agent succeeds on task completion but fails on recovery, or this model answers correctly but uses twice as many low-value retrieval rounds as the baseline.

Start with traces from the work your users actually care about. For each task family, define the events that must happen before a completion claim is allowed. For a coding agent, that might be file inspection, a targeted edit, and a test or type check. For a research agent, it might be source diversity, evidence extraction, and a synthesis checkpoint. For an operations agent, it might be approval verification before any write action.

Then run the same trace checks across prompts, models, tools, and agent frameworks. Keep the scoring dimensions stable enough that regressions are meaningful. If a new model improves answer quality but doubles unsupported completion claims, that is not a simple upgrade.

A strong agent trace reads like disciplined work. The agent samples reality, names uncertainty, uses tools selectively, incorporates evidence, adapts when something fails, and verifies before declaring completion. It is not necessarily the shortest trace. It is the trace where each step has a reason and the final answer is supported by what happened.

That is the bar teams should evaluate against before shipping agentic systems into customer-facing or operational workflows.