Encode your AI evaluation release gate in CI

A release gate is a decision rule, and the first thing it needs is a statistic that does not move when nothing changes. Agent evals behave as repeated-trial experiments rather than single-shot scores, and Anthropic's engineering guidance documents running multiple trials per task for that reason, with per-task success rates that can sit near 90% on one task and 50% on another, so a task that passes one run fails the next.^[1] If your suite reports one number per run and that number drifts on an unchanged commit, the gate has no signal to block on yet.

Choose between the three candidate statistics deliberately, because they diverge as the trial count grows. Anthropic separates pass@k, the probability that at least one of k attempts succeeds, which rises with k, from pass^k, the probability that all k trials succeed, which falls with k, and gives the arithmetic: 75% per-trial success over three trials is roughly 42% pass^3.^[1] At k=1 the two coincide; by k=10 they tell opposite stories, and Anthropic recommends pass^k for consistency-critical customer-facing agents while coding work usually gates on pass@1.^[1] Arize lists the same three statistics and warns that stochastic behavior can make a weak agent look reliable when the dataset or run count is too small.^[2]

Then split the suite by role, because a single threshold cannot serve both purposes. Anthropic separates target pass rates by suite purpose: capability evals should start at a low pass rate to leave a hill to climb, while regression evals should sit at nearly 100%, so a drop signals breakage.^[1] Gate the regression suite and report the capability suite. Isolation matters at the same time — Anthropic reports that shared state such as leftover files and caches creates correlated failures, and that Claude once gained an advantage by reading git history from earlier trials.^[1]

Agreed targets in a spreadsheet are not a gate. The mechanism that enforces them is the test runner your CI already understands. DeepEval documents that every metric takes a passing threshold plus strict_mode and include_reason, and that the CI path runs through assert_test() and the deepeval test run command, layering evaluation features onto pytest so that failing metrics fail the build and agent regressions get caught before they ship.^[3] TypeScript suites use a Vitest toPass() matcher for the same effect.^[3]

The wiring is three steps in the order that removes hand-maintained fixtures: build an EvaluationDataset of Golden objects, instrument the agent with @observe plus update_current_trace so test cases are constructed from the trace automatically, then parametrize over the goldens and assert.^[3] Where each metric is attached matters as much as its threshold. DeepEval prescribes placing tool-correctness and argument-correctness strictly on the LLM component that makes the decision, while trajectory metrics need the full ordered trace.^[3]

Add the pull-request layer so the verdict lands where the change is reviewed. Braintrust describes a GitHub Action that runs evaluations on every pull request, posts results as comments, and blocks the pull request on a quality drop, though that page defines no score cutoffs or configuration syntax.^[6] Databricks frames the same control as promotion: thresholds define acceptable behavior and act as gates for promotion to staging or production, with deployment gates requiring new versions to clear threshold-based checks before rollout, and it recommends applying the same criteria across dev, staging, and production while restricting who may change evaluation criteria or promote a version.^[4] Two gaps you have to build yourself: DeepEval's page publishes no numeric defaults and documents no warn-only mode, flake retries, or score-delta gating.^[3]

A grader that reads the final message grades a sentence. Anthropic defines the outcome as the environment's final state rather than the agent's claim: a booking agent may announce success, but what counts is whether a reservation exists in the environment's SQL database, with documented end-state assertions including tickets at status resolved, refunds at status processed, and a security log event auth_blocked.^[1] The same guidance points at how established harnesses check this — WebArena inspects URL and page state plus backend verification that an order was actually placed, and OSWorld inspects filesystem, configs, database contents, and UI properties.^[1]

Arize is explicit that the acceptable final state may live in a database, ticketing system, codebase, generated document, or physical environment, and that the agent's confirmation message cannot establish it: the evaluator inspects the affected systems after execution against explicit completion criteria, ideally with deterministic assertions.^[2] Its refund case requires three verified conditions — the correct order refunded at the approved amount, the customer record updated, and the case removed from the unresolved queue — and treats partial completion as a distinct verdict, since a recorded refund with an open case is not the same as no state change.^[2] It cites Terminal-Bench, where hidden tests inspect the resulting environment rather than the final response.^[2] IBM makes the same point for agents that emit no text at all: updating a record or sending a message is measured by correct execution.^[5]

Two design rules keep these checks usable in CI. Bound the episode with transcript-level limits such as max_turns: 10, so a non-terminating run produces a definite failure instead of a hung job.^[1] And grade the product, not the path — Anthropic warns that requiring exact tool-call sequences is too rigid and produces brittle tests, so a reference path should define required actions and boundaries rather than one mandated sequence.^[1]

A release where task completion improved and the agent took an unauthorized action on two runs should not pass. Arize states that reliability thresholds should reflect both failure frequency and operational consequence, because an average success rate can mask rare unauthorized actions, and describes the gate shape directly: a candidate may need to exceed a required task-completion and policy-compliance rate while remaining below an allowed critical-failure rate.^[2] Write the gate as a conjunction and a single guardrail breach fails the build no matter what the primary metric did.

Arize's scorecard separates four roles: the primary outcome (verified completion, resolution rate, test pass rate), guardrails (critical-failure rate, policy violations, unauthorized actions), diagnostics (tool accuracy, grounding, escalation quality, retry rate), and operational limits (cost per successful task, P95 latency, tokens per successful task) — and it says a rising aggregate alone is insufficient, since the primary outcome must improve without breaching a guardrail or a cost or latency limit.^[2] It also warns against relying on aggregate scores and recommends breaking results out by task type, tool, risk level, and failure category, which is what lets the gate hold tighter budgets on high-risk slices than on the suite as a whole.^[2]

Structured tool-call failures deserve their own countable budget. IBM supplies a rule-based function-calling taxonomy you can count against a ceiling: wrong function name, missing required parameters, wrong parameter value type, a value outside the allowed set, and hallucinated parameters not defined in the function specification.^[5] Add the inverse tripwire as well, because a red result is not always a real failure: Anthropic notes that a 0% pass rate across many trials, such as 0% pass@100, is most often a signal of a broken task rather than a broken agent, so keep a reference solution proving solvability.^[1]

Absolute cutoffs either block every release or never fire once the dataset starts changing. Arize's loop is delta-based: production trace, confirmed failure, regression case, experiment, deployment, with each change to prompt, model, tool, or orchestration run as a separate experiment that holds dataset, evaluators, and run config fixed, through the stages Baseline, Candidate, Compare, Investigate, Record.^[2] Pin the baseline as a tuple — version, dataset version, evaluator set, run config — and compute per-case deltas against it.

The blocking condition that reads cleanly in review is the flip set. Arize prioritizes examples that flipped pass to fail and locating the first point where the trace diverged, so the gate can block on any previously passing gated case that regresses and hand the reviewer a divergence point instead of a score drop.^[2] Braintrust describes the tooling shape for this: baseline comparisons with per-test-case score deltas, versioned datasets that track performance across changes, failed experiments joining the CI regression suite, and one-click promotion of a production failure into a permanent regression test — with no score cutoffs or gating syntax published on that page.^[6]

Databricks adds the surrounding controls: record a baseline, set improvement targets, loop test, evaluate, refine, and retest, and version datasets so comparisons are reproducible.^[4] Record which run drove which promotion decision, or a delta comparison stops being reproducible the moment someone regenerates the dataset.

A gate is only as strong as the cases behind it, and the best source is the failures you have already seen. Arize specifies the regression-case fields to capture — input, relevant_context, available_tools, expected_behavior, observed_failure, failure_category, source_trace_id — and requires hygiene before promotion: confirm the failure, drop duplicates and sensitive data, keep the minimum state needed to reproduce, run the fix against the case before merging, then retain it in the suite.^[2] That last step is what makes the case a gate rather than a ticket.

Suites also need a graduation path so effort is not spent re-proving solved work. Anthropic notes that saturated capability evals can graduate into the continuously run regression suite, which is the same suite your gate blocks on.^[1] Size the starting set for the effect you need to detect: begin with 20 to 50 simple tasks drawn from real failures and grow the suite as the agent matures and effect sizes shrink.^[1]

On the execution side, keep promoted cases in the same representation the runner already consumes — an EvaluationDataset of Golden objects that test cases are parametrized over, so adding a case is a data change rather than new test code.^[3] Databricks recommends a documented failure taxonomy and versioned datasets for exactly this reason, so comparisons across agent versions and architectures stay meaningful as the case bank grows.^[4]

A green build is evidence about a fixed dataset under a fixed configuration, and it is not the same as evidence about users. Databricks recommends confirming offline gains with A/B tests using randomized sampling and confidence intervals, because offline wins can hide regressions that only appear with real traffic.^[4] For a major model, prompt, or orchestration change, treat the CI gate as necessary but not sufficient, and run the same evaluators against production traces after rollout.^[2]

Audit the grader before you trust a red or a green verdict. Over-rigid string matching, under-specified prompts, inverted thresholds, and one-sided datasets that reward over-triggering all produce false failures, and Anthropic's warning about brittle exact-sequence checks belongs in the same review.^[1] IBM's function-calling taxonomy is useful here too, because a run can be marked wrong for a parameter-type mismatch that no user would notice, or marked right despite a hallucinated parameter the specification never defined.^[5]

Finally, keep the reporting surface wide enough to explain a block. Arize recommends breaking results out by task type, tool, risk level, and failure category rather than leaning on aggregate scores, and Anthropic recommends reporting per-task pass rates alongside the aggregate.^[2][1] A gate that fails without naming the slice, the case, and the divergence point will be waived the first time a release is urgent.