Building evaluations from production traces

Synthetic evals are useful for coverage, but production traces show where the system actually meets the business. They capture messy inputs, partial context, tool latency, permissions, missing data, ambiguous instructions, and user behavior that no clean prompt set will fully anticipate.

The best source material is not only catastrophic failure. Escalations, manual corrections, long-running traces, abandoned sessions, near misses, and expert overrides all reveal places where the agent needs clearer constraints or better recovery.

Production traces often contain private data. Redaction should remove sensitive values while preserving the relationships that matter for evaluation. If the task depends on a join key mismatch, the redacted trace still needs mismatched keys. If the failure depends on a policy exception in free text, the redacted text still needs the exception shape.

A good redaction process keeps event order, tool type, error type, evidence markers, approval requirements, and final answer structure. Replace secrets and user identifiers, but do not flatten the trace into a generic prompt.

A trace becomes an eval when it has an explicit contract. The contract should define the user goal, required evidence, allowed and forbidden tools, success criteria, budgets, and known failure modes. This lets the same case run repeatedly across models, prompts, or agent frameworks.

Do not encode only the final answer. Encode the behaviors that must happen along the way. If the original incident happened because the agent skipped a policy document, the eval should require evidence from that document. If it failed because approval was bypassed, the eval should include an approval precondition.

Not every check should be automated. Deterministic checks are excellent for presence, order, budget, forbidden actions, and exact artifacts. Expert review is better for whether a recovery strategy was appropriate, whether an answer was sufficiently nuanced, and whether the agent respected business context.

The right pattern is layered. Run deterministic checks first so reviewers spend time on judgment rather than bookkeeping. Then capture expert decisions in structured labels so the suite improves over time.

This whole approach has a precondition that teams discover late: the traces you are already collecting may not contain the prompts and completions you need. The OpenTelemetry generative AI semantic conventions, which moved out of the main semantic-conventions repository into a dedicated one, define the message-body attributes gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, and gen_ai.tool_definitions as opt-in.^[1] A default-configured setup records structure and token usage but not the content.

That means a team can run GenAI tracing for a year, decide to build evals from production traces, and find that every stored span has the shape of the conversation and none of its substance. Turning content capture on is a deliberate decision with privacy consequences, which is exactly why it should be made before the incident you want to learn from rather than after. Enable it with a redaction path already designed, not as an afterthought.

Adopting the standard attribute names is still worth doing, with one caveat stated plainly: every gen_ai attribute in these conventions is currently Development stability, so the naming is not settled and should not be described as an industry standard.^[1] Structural attributes such as gen_ai.operation.name, gen_ai.agent.name, gen_ai.conversation.id, and the usage counters gen_ai.usage.input_tokens and gen_ai.usage.output_tokens give you a portable vocabulary today, and pinning the convention version you wrote against protects you when the names move.

A production-derived eval suite is not a one-time project. It should change as the product changes. New tools add new failure modes. New models change behavior. New customers bring new edge cases. Treat the suite like a reliability asset with ownership, review cadence, and release impact.

The most useful suites become part of the operating rhythm. They run before model upgrades, prompt changes, retriever changes, and agent framework changes. They also run after incidents to confirm the failure has become a durable regression test.

A reasonable starting size is smaller than most teams expect. Anthropic's agent-evals guidance suggests that 20 to 50 simple tasks drawn from real failures is a strong start, and argues that maintaining evals should be as routine as maintaining unit tests.^[2] The cadence matters more than the initial count, because a suite with no additions this quarter has started drifting away from the product it is supposed to guard.