Benchmark pages should show the work

A benchmark page should start with the work, not the score. What is the agent trying to build, fix, prove, retrieve, or decide? What inputs does it receive? What tools can it use? What is forbidden?

For a Vectory proof-grounded policy task, the public page should show a synthetic task like proof_grounding.policy_proof.001: answer a policy question using provided source references, preserve a warning field, close three obligations, and pass two checker results. That is concrete enough for a reader to judge the benchmark.

The environment defines what the score means. A code benchmark with internet access is not the same as one with a locked container. A retrieval benchmark with a clean corpus is not the same as one with conflicting documents. A policy benchmark with trusted checker commands is not the same as one where the submission can define its own grader.

Vectory reports should name the environment contract: workspace path or fixture version, suite manifest version, tool policy, timeout, model budget, and whether formal runtime was disabled, artifact-only, or explicitly enabled.

A score should be decomposed into parts that match the behavior under test. One number is useful for sorting, but it cannot explain whether the agent failed because of retrieval, evidence, proof grounding, recovery, cost, or artifact correctness.

A stronger page shows the score recipe. Example: proof_grounding 30 percent, task_completion 25 percent, verification_integrity 20 percent, recovery 15 percent, cost_control 10 percent. Then it lists the exact checks that moved each dimension.

The best benchmark pages let readers inspect artifacts: the submitted trace, checker output, score JSON, failure cases, and a minimal repro. Without artifacts, the page asks readers to trust the author rather than the evaluation.

Vectory should include both a passing sample and a failing sample. The failure sample is often more persuasive because it proves the grader catches the shortcut: an open proof obligation, a failed checker ignored by the final answer, or evidence that does not support the claim.

For long-horizon work, a final score hides the most important question: did the agent make measurable progress or just produce more text? Checkpoint curves catch stalls, regressions, and late unsupported success claims.

A Vectory benchmark page should display score by checkpoint, artifacts created by checkpoint, failed checks by checkpoint, and the first moment where the final answer became supported. This makes progress auditable instead of theatrical.

The single most common omission on a benchmark page is a range. Terminal-Bench 4.0 publishes resolution rate beside cost and tokens and states that the whiskers on its chart span the 95 percent confidence interval, so a reader sees a spread rather than a point.^[1] METR goes further and fits success probability against human task length per model, drawing 95 percent confidence bands from hierarchical bootstrapping over task families, tasks, and attempts.^[2] Neither approach requires unusual machinery. Both simply refuse to present a sample as a measurement.

The second omission is the grader's own error rate. Anthropic's evaluation guidance contrasts a vague success criterion such as safe outputs against a measurable one stating less than 0.1 percent of outputs out of 10,000 trials flagged for toxicity by the content filter.^[3] The second version names a population, a trial count, and a detector, which is what lets two teams run the same test and compare answers. A page that publishes a threshold without naming the detector has published a preference, not a criterion.

The third and rarest disclosure is where the evaluation itself failed. Anthropic extended its evaluation cadence from three months to six in version 2.0 of its Responsible Scaling Policy to avoid lower-quality, rushed elicitation, and its self-review disclosed concrete shortfalls including evaluations finished three days late and missing elicitation techniques such as best-of-N and chain-of-thought.^[4] That is the model to copy. A gate that never reports its own gaps gives a reader no way to judge how much the passing runs are worth, and a benchmark page that shows only wins is advertising.

None of this makes a page longer than it should be. A confidence interval is one column. A detector name is one clause. A known-gaps section is a short list. The cost of publishing them is small, and the alternative is asking readers to trust an author instead of an evaluation.