Evaluate tool-call accuracy as four separate checks

A single tool-use score tells you the agent did something wrong with its tools and nothing more. Vendor guidance has moved away from that aggregate. Arize's agent-evaluation guide splits router scoring into tool or skill selection, graded against an expected route, and parameter extraction, graded on schema validity, grounding in the request, and completeness, and it names the diagnostic payoff directly: repeated confusion between the same two tools points at overlapping tool descriptions, while the right tool with bad arguments points at the schema or the extraction logic.^[1] DeepEval ships the same split as two distinct component-level metrics, ToolCorrectnessMetric against an expected tool list and ArgumentCorrectnessMetric over the generated arguments.^[4]

Add ordering and recovery and you have four checks answering four different questions: did the agent reach for the right capability, did it fill the call correctly, did it sequence the calls in a valid order, and did it do something sensible when a call came back as an error. Databricks treats trajectory evaluation as exact match plus in-order and any-order matching of required steps, with precision and recall of essential actions and efficiency measures that catch loops or redundant steps, which makes ordering a per-step property rather than one equality test.^[3] Arize describes recovery and escalation as their own evaluator, examining errors, retries, stop conditions, fallbacks, and escalation events.^[1]

Each check needs its own reference data, so the expected trace has to be structured for four readers rather than one. That is a case-authoring requirement more than a dashboard requirement: if a test case stores only a golden transcript, nothing downstream can separate a mis-route from a bad argument, and nothing can distinguish a wrong order from a valid alternate path. Write the expected trace as named fields, one per check, and let each grader read only the field it owns.

Score selection first, because an argument score computed over mis-routed calls is noise. Arize recommends a deterministic exact match against the expected route, aggregated as per-route accuracy with a confusion matrix, and reserves an LLM judge for cases where several routes are genuinely defensible.^[1] The matrix is the part worth insisting on: a single accuracy figure says selection got worse, while the matrix says which pair of tools the model keeps swapping, which is an actionable edit to two tool descriptions.

DeepEval gives this a concrete implementation. ToolCorrectnessMetric compares the tools actually invoked against an expected_tools list, and its evaluation_params control whether that comparison covers only tool names or also input parameters and outputs, so a run can be scored on routing alone while argument grading stays in its own metric.^[4]

Then build the selection set from cases that can actually fail. Arize lists the hard ones: overlapping tool descriptions, missing required parameters, requests that need no tool at all, multi-turn context, and misleading identifiers.^[1] A suite of unambiguous single-tool requests will report high selection accuracy and catch none of the failures that reach production, so treat hard-case coverage as part of the metric definition rather than a later addition.

Grade arguments on the subset of cases where selection passed, so a mis-route never silently depresses the argument score. Arize scopes parameter extraction to schema validity, grounding in the request, completeness, and field-level precision and recall, which keeps the result readable at the field level instead of as one pass or fail for the whole call.^[1] Field-level reporting is what tells you that one enum field fails constantly while the rest of the schema is fine.

Make the failure modes assertions with names rather than a numeric penalty. IBM enumerates the rule-based argument failures worth asserting on directly: wrong function name, missing required parameter, type mismatch, out-of-range value, and hallucinated parameters absent from the specification.^[5] Each routes to different work. A type mismatch is usually a schema description problem, an out-of-range value is a validation and prompt problem, and a hallucinated parameter is a sign the tool specification the model sees does not match the one the runtime enforces.

Check the default behavior of whatever metric you adopt before trusting a green result. In DeepEval, ArgumentCorrectnessMetric evaluates the generated arguments as its own component-level metric, while ToolCorrectnessMetric compares only tool names unless evaluation_params is set to include input parameters, so an argument regression can pass a tool-correctness check that was never configured to look at arguments.^[4]

Exact-sequence assertions are the fastest way to build an eval suite that fails on correct runs. Anthropic calls the instinct to check a specific tool-call order too rigid and a cause of brittle tests, because agents regularly find valid approaches the eval designer did not anticipate, and recommends grading what the agent produced over the path it took.^[2] Arize makes the compatible point for reference-based comparison: references should define required actions and boundaries rather than mandate one exact sequence.^[1]

Keep ordering as a scored dimension anyway, because some sequences are genuinely wrong. Databricks describes trajectory evaluation as exact match plus in-order and any-order matching of required steps, with precision and recall of essential actions reported separately, which is exactly the control you need: attach in-order matching to the constrained pairs and any-order matching to everything else.^[3] Anthropic's support example shows what a load-bearing constraint looks like in practice, verifying identity before processing a refund.^[2]

Treat loops and redundant steps as their own assertion rather than a side effect of the order check. Both Arize and Databricks list redundant steps and loops as a distinct trajectory defect, and Arize quantifies step efficiency as path convergence, C = (1/n) × Σ (s_min / s_i), bounded by 0 < C ≤ 1, while cautioning that the shortest observed run is only a proxy for the optimal one.^[1][3] Pool runs of the same task type to fix a shared s_min, and read convergence next to task completion rather than as a quality score on its own.

When a sequence assertion fails, the first question is whether the agent was wrong or the grader was. Anthropic identifies reading the transcript to decide whether the agent made a genuine mistake or the grader rejected a valid solution as the key triage step for path-based tests.^[2] Build that triage into the workflow: an ordering failure that turns out to be a valid alternate route is a case-authoring fix, so edit the constraint instead of waiving the result run by run.

Recovery only becomes measurable once failures are deterministic, so put the faults in the dataset. Add cases where specific tools return error status codes, empty results, and malformed payloads, then assert the mechanical contract with code checks. Arize describes a recovery and escalation evaluator that examines errors, retries, stop conditions, fallbacks, and escalation events, pairing code checks for retry ceilings and mandatory escalations with LLM judges for recovery quality.^[1] Those code checks are the cheap half and they catch the two worst behaviors: no retry at all, and a retry loop that never stops.

Separate reading the result from recovering from it. Arize treats tool-result interpretation as its own evaluator, whether the agent understood and acted on what came back, using a judge plus deterministic checks on status codes or returned values, on the grounds that selecting the wrong tool, generating bad arguments, and misreading a response are independent failures.^[1] Databricks reinforces the same split by scoring successful execution and correct interpretation of outputs separately from selection and argument accuracy, and notes that errors often originate several steps upstream of the final output.^[3]

Then refuse to accept the agent's own report as evidence. Anthropic's backstop is to verify the environment state, such as whether a reservation exists in the SQL database, rather than the agent's claim of success.^[2] Without that assertion, a confident summary after a failed call reads as a pass. For consistency, Anthropic points to pass^k, the probability that all k trials succeed, and Arize lists recovery consistency as a reliability dimension measured alongside pass@1, pass@k, and pass^k.^[2][1] Run the injected-failure cases repeatedly and report that figure, because recovery that works once is not a property you can ship.

Finally, label each recovery failure by stage so the result names the step. No retry, retry loop, wrong fallback, misread result, and false success claim are five different tickets, and a blended trajectory score collapses them into one downward line.

Aggregation is a separate decision from scoring, and it should be written down. Anthropic sets out the options plainly: grader results may be combined as weighted, where combined scores must hit a threshold, as binary, where all graders must pass, or as a hybrid.^[2] Pick per test case and record the choice. Binary suits correctness-critical calls such as a refund, where a wrong argument is not partially acceptable; weighted suits exploratory tasks where an ordering deviation is not a defect.

Give composite tasks partial credit, and keep judges narrow. Anthropic recommends building in partial credit because an agent that diagnoses the issue and verifies identity but misses the refund is meaningfully better than one that fails immediately, and advises grading each dimension with an isolated LLM-as-judge rather than one judge scoring every dimension at once.^[2] It also pairs code-based tool-call verification of the tools used and their parameters with transcript analysis of turns taken and token usage, while noting that such graders are brittle to valid variations.^[2] That caveat is the reason the four component scores stay visible: a brittle grader is tolerable when a reader can see which check it was.

DeepEval enforces the split structurally, which helps if you want the framework to hold the line for you. Component metrics such as ToolCorrectnessMetric and ArgumentCorrectnessMetric attach to the span through the observe decorator, while trajectory metrics including TaskCompletionMetric, StepEfficiencyMetric, PlanQualityMetric, and PlanAdherenceMetric run through the evals iterator and must be used with tracing enabled, so the execution-level metrics cannot run standalone.^[4] Emit all four component scores per run regardless of framework, and make the aggregate unreadable without them.

The checks earn their keep when a pipeline fails with the name of the grader that dropped. LangSmith supports evaluators that score intermediate decisions and agent behavior, custom evaluators written in Python or TypeScript, and CI runs through pytest, Vitest, and GitHub workflows with thresholds that fail pipelines on score drops.^[6] Give each of the four checks its own threshold in that configuration rather than one combined gate, so the failure message carries the specific grader rather than a score delta someone has to investigate.

Keep the dataset side maintained alongside the thresholds. The selection set needs the hard cases, the argument set needs the named failure modes, the ordering set needs constraints written only where order is load-bearing, and the recovery set needs injected faults with enough repeats to report consistency. IBM's enumeration of rule-based argument failures is a convenient checklist for the second of those, since each item maps to one assertion you can write once and reuse across tools.^[5]

One habit makes the whole arrangement stable: when a check fails, fix either the agent or the case, and never the threshold. Anthropic's triage question, whether the agent made a genuine mistake or the grader rejected a valid solution, is the right first read on every red result, and both answers lead to a concrete edit rather than a loosened gate.^[2]