How to fully evaluate software engineering agents

A useful software-agent eval starts before the model runs. The task contract should state the repo state, allowed tools, expected artifact, required checks, budget, and stop condition. Without that contract, a polished final answer can hide skipped work.

For example: task_id software.fix_rag_timeout.004; goal reduce the timeout path without changing public API; required evidence pytest tests/test_retriever_timeout.py, pnpm build, and one trace event showing the agent inspected the retry policy before editing.

The trace tells you whether the agent did the work it claims to have done. Vectory should score events such as file inspection, search rounds, command execution, edits, verification, and final answer claims. This catches a common failure: the model writes a confident summary after touching the wrong file or never running the relevant test.

A minimal trace record is enough to make this concrete: events^[3].type = command, events^[3].name = pytest, events^[3].status = failed; claims[0].text = timeout fix is complete; claims[0].evidence_ids = [checker.pytest.timeout]. That run should be penalized unless a later checker closes the failure.

High-quality agent evals need executable graders. For code tasks, that can be tests, type checks, lints, browser screenshots, API probes, or policy checkers. For UI tasks, a screenshot or DOM assertion should sit next to the final answer. For proof-grounded tasks, every required obligation needs a closed checker result.

A strong Vectory run report should say: vectory_score 0.91; proof_grounding 0.94; recovery 0.82; pathologies none detected. A weak but plausible run should say: vectory_score 0.58; ignored_checker_result 1; unsupported_completion_claim 2.

Long-horizon benchmarks need time-series scoring. A 24-hour agent run that reaches 70 percent after two hours and then regresses is different from one that steadily improves. A run that burns tokens while the score stays flat is also a product signal.

Vectory should report checkpoint rows: t+15m score 0.22 with open obligations 5; t+60m score 0.61 with failed checks 2; t+120m score 0.88 with failed checks 0. The curve explains whether the agent converged or merely produced activity.

The public software-agent benchmarks already demonstrate most of what an internal report should carry, and copying their disclosures is cheaper than inventing a format. SWE-bench Verified is a human-validated subset of 500 SWE-bench instances, built with OpenAI, where annotators checked that problem statements were understandable, test patches valid, and tasks achievable.^[1] That validation pass is the work, not overhead on top of it, and it is the reason a task-quality review belongs in your own suite before any score is trusted.

Terminal-Bench 4.0 shows the second half of the discipline. Its leaderboard reports resolution rate beside cost and tokens, and it states that the whiskers on its chart span the 95 percent confidence interval.^[2] Both choices matter for a release decision: a quality win the product cannot afford is a business decision rather than an upgrade, and a point estimate with no interval cannot distinguish a regression from sampling noise. The same page carries a canary string declaring that its data should never enter training corpora, which is a concrete contamination control worth imitating for any private suite.

Harness identity belongs in the same disclosure. SWE-bench states that results from mini-SWE-agent release 1.x and 2.x are not necessarily comparable, because 1.x parses actions from output strings while 2.x uses tool calling, and temperature was pinned at 0.0 for 1.x and left unset from 2.x onward.^[1] If a public benchmark has to warn readers that two of its own numbers describe different experiments, an internal report comparing last quarter to this one needs the same warning.

For long-horizon work, be careful about how a trend is stated. METR fits success probability against human task length per model and draws 95 percent confidence bands from hierarchical bootstrapping over task families, tasks, and attempts, reporting a 50 percent time horizon that has been doubling roughly every seven months with stated uncertainty of about one to four doublings per year.^[3] Cite the caveat with the figure: that page states some of its text and figures are out of date, notes that only the interactive chart is maintained, and points to Time Horizon 1.1, announced 29 January 2026.

The goal is not a more dramatic leaderboard. The goal is a boring release decision: this agent can ship for these tasks, under these budgets, with these remaining pathologies. That is a stronger output than a single pass rate.

A team should be able to open a Vectory report and decide whether to promote a model, block a prompt change, require more human review, or add a regression task from the failure that just appeared.