Instrument agent traces so they can become evaluation cases

The production run you most want as a regression test is usually the one you cannot replay. The trace captured what the agent said and which model calls it made, but not the context it was given or the tools it could choose from, so rerunning it produces a different situation rather than the same test. Arize's regression rows fix that by carrying input, relevant_context, available_tools, expected_behavior, observed_failure, failure_category and source_trace_id, and by preserving the smallest amount of state required to reproduce and judge the case.^[1]

Anthropic draws the matching line inside a single run: the transcript is the full record of one trial, including outputs, tool calls, reasoning and intermediate results, which for the Anthropic API is the complete messages array.^[2] DeepEval makes the shape executable with spans typed as agent, llm or tool and a trace-level input and output set by update_current_trace, so a pytest run builds each test case from the trace instead of requiring manual extraction of inputs and outputs.^[3]

The practical consequence is that the envelope has to be decided at instrumentation time, not at promotion time. Mint the stable trace id at ingest, because Arize's regression rows key back to the originating run through source_trace_id and a reconstructed identifier cannot be trusted to point at the run you reviewed.^[1] The field names below illustrate record shape; they are not measured results.

A replay with a different toolset is a different test. That is why Arize keeps available_tools in the row schema alongside the context, and why an evaluation harness that resolves tools from current configuration silently changes the case every time someone ships a new integration.^[1] Pin the inventory the run saw, then pin the versions of anything whose behavior the grader depends on.

Arguments matter as much as names. DeepEval compares invoked tools and their arguments against expected_tools, and records tool spans with the tools actually called and the arguments passed.^[3] Anthropic lists tool-call verification, meaning which tools ran and with what parameters, among the code-based grading methods worth using.^[2] Neither check is possible from a log that stores only a list of tool names.

Then make the record readable as a unit. Arize notes that production signals are usually split across model calls, tool runtimes, APIs and client infrastructure, and proposes a blunt test: can an engineer reconstruct the run without stitching scattered logs together.^[1] If the answer is no, join the telemetry before you store the case, because the stitching work will not survive the person who did it.

The agent's own report of success is not evidence. Anthropic separates the transcript from the outcome and makes the distinction concrete with a flight-booking example: the check is whether the reservation exists in the environment's SQL database, not whether the agent announced that it booked one.^[2] So the trace record needs a place for post-run environment state, written by the harness rather than parsed out of the final message.

Attach the cheap counters while you are there. Anthropic's sample task definition tracks n_turns, n_toolcalls and n_total_tokens, and notes that latency, token, cost and error-rate baselines come for free once you run a fixed task bank; Arize spans similarly carry latency, cost and state changes alongside inputs and outputs.^[2][1] Those baselines only exist later if the per-run record captures them now.

Store a reference solution with the case as well. Anthropic uses reference solutions to confirm that a task is solvable and that graders are configured correctly, which is what distinguishes a genuine red result from a grader rejecting a valid answer.^[2] Without it, every failing case starts with an argument about whether the case is even fair.

Trajectory scoring is where trace-derived suites usually go brittle. Anthropic reports that checking whether agents followed specific ordered steps proved too rigid and results in overly brittle tests, because agents find valid paths nobody anticipated, and recommends grading what the agent produced rather than the path it took.^[2] Order-sensitive assertions belong to the small set of tasks where the order is the requirement.

Instrumentation coupling is the second failure mode. Arize warns that evaluators tied to a specific tool name, span layout, or vendor-specific trace field break on refactors, and that checks on resulting state, policy conditions, or normalized events are more durable; its evaluators can run at span, trace, session, or experiment level, so the scope is a decision per criterion.^[1] Emit normalized events next to raw spans and point the durable checks at those.

Component checks need the right span. DeepEval splits trajectory metrics that read the full ordered trace, including PlanAdherenceMetric, TaskCompletionMetric and StepEfficiencyMetric, from component metrics such as ToolCorrectnessMetric and ArgumentCorrectnessMetric that attach to a single span and belong on the LLM component, since that is where tool choices and arguments originate.^[3] Attach them elsewhere and the metric reads a layer that never made the decision.

For anything a judge has to read, split the dimensions. Anthropic recommends a separate isolated judge per dimension, specific rather than vague rubric criteria, an Unknown escape hatch to curb hallucinated verdicts, calibration against human experts, and partial credit on multi-part tasks.^[2] Each of those needs the trace to expose the relevant artifact on its own, not buried in one concatenated blob.

Promoting traces wholesale means customer data lives in your fixtures indefinitely. Arize describes the promotion step as a curation loop rather than an export: filter traces for failed evaluations, pick representative examples rather than exporting everything, confirm the failure, drop duplicates and sensitive data, preserve the smallest amount of state required to reproduce and judge the case, attach the expected outcome or human label, and route ambiguous or high-risk cases through human review before collection.^[1] Minimizing at write time is cheaper than scrubbing a span tree later.

Encode the selection as rules so the retained set is reviewable. Arize's auto-add rules collect spans by evaluation label, latency, token usage, or tool activity, which turns promotion into a policy someone can audit instead of a series of ad-hoc exports.^[1] LangSmith supplies the human path as infrastructure: annotation queues that flag runs, automatically send interesting traces for expert review, embed UI renderings in the review flow, and standardize reviewer feedback with shared scoring criteria, with judge output as a starting label refined by human annotation.^[4]

Match the storage boundary to the sensitivity of what you keep. Galileo's framework comparison notes self-hosted and hybrid deployments positioned for data sovereignty, open-source cores that can run inside an existing security perimeter, and audit logging with immutable records.^[5] Decide that boundary before the first promoted fixture, because moving a case bank after it fills with production data is a migration nobody schedules.

If the dataset and graders move with the agent, a score change tells you nothing. Arize's stated path runs from production trace to confirmed failure to regression case to experiment to deployment, with fixes rerun against fixed datasets and evaluators so experiment comparisons stay valid.^[1] Databricks adds versioned datasets with Unity Catalog lineage for reproducibility, plus deployment gates on threshold checks.^[6]

Separate the buckets so a drop is interpretable. Anthropic distinguishes regression evals, where the agent should still handle everything it used to and scores are expected near 100 percent, from capability evals that start low and later graduate into the regression suite once they pass.^[2] Stamp a failure_category on every promoted trace as well: Arize includes it in the regression row, and Databricks reports that a documented failure taxonomy supports more reliable comparisons across agent versions and architectures.^[1][6]

Wire the comparison into the pipeline rather than a notebook. LangSmith integrates with pytest, Vitest and GitHub workflows with thresholds that fail pipelines when scores drop, and provides side-by-side experiment comparison views; Galileo cites a native GitHub Action that surfaces evaluation results on every pull request.^[4][5] Databricks then confirms offline deltas in production with randomized A/B sampling and confidence intervals.^[6]

Finally, watch for a bank that has stopped discriminating. Anthropic warns that saturation can mask real gains and cites Qodo, whose one-shot evals did not capture the gains on longer, more complex tasks.^[2] When two versions score the same on a frozen bank, the next move is harder cases promoted from recent traces, not a conclusion that the change did nothing.