Unfalsifiable stories of agent progress

Claims like the agent is production ready or the new model is better are not engineering statements. They are narratives. A team cannot use them to decide whether to ship, rollback, or add review.

A falsifiable Vectory claim looks like this: on suite agent.release_readiness.2026_08, the agent must score at least 0.86 overall, at least 0.90 on proof_grounding, produce zero critical pathologies, stay under 18,000 tokens, and close every required obligation.

A claim without evidence requirements can pass by luck. The agent might answer correctly while using the wrong source, skipping a checker, or hiding uncertainty. That is why Vectory should make evidence part of the claim itself.

For example: claim_id final.policy.allowed must link to evidence_ids source.policy.17 and checker.supporting_rules.passed. If the final answer cites source.policy.12 instead, the text might sound right, but the claim is unsupported.

An aggregate score can improve while the system becomes riskier. A model may pass more tasks but also ignore more failed checkers. Another may answer faster by skipping retrieval. Pathology budgets prevent that trade from hiding inside the mean.

A Vectory release gate should say: pass if vectory_score >= 0.86, unsupported_completion_claim = 0, ignored_checker_result = 0, policy_regression = 0, placeholder_proof_accepted = 0, and search_without_convergence <= 2.

Better is not one dimension. An agent can improve retrieval and regress recovery. It can reduce cost and weaken evidence. It can pass more tasks while making human review harder. A useful report names which behavior changed.

The right comparison is a delta table: task_completion +0.04, proof_grounding +0.11, recovery -0.07, mean_tokens +18 percent, critical_pathologies unchanged. That gives a release owner something real to decide.

A credible benchmark report should make failure inspectable. If the agent failed, show where. If it passed, show what would have made it fail. Otherwise the report becomes another story about progress.

Vectory should treat the failure surface as first-class output: failed task IDs, event IDs, unsupported claims, open obligations, checker summaries, and a short recommendation for the next suite addition.

This argument is easy to make and harder to live with, so it helps to point at organizations that publish gates against themselves. Anthropic's Responsible Scaling Policy ties a Capability Threshold to a required standard before further scaling, and its version history records the kind of specificity that makes a commitment testable: the evaluation cadence was extended from three months to six in version 2.0 to avoid lower-quality, rushed elicitation, and version 3.1 pinned the AI research and development threshold to compressing two years of 2018 to 2024 AI progress into a single year in aggregate capability terms rather than researcher productivity terms.^[1]

The more instructive part is the self-reporting. Anthropic's own review disclosed shortfalls including evaluations finished three days late and missing elicitation techniques such as best-of-N and chain-of-thought.^[1] A gate that publishes where it fell short is doing something a marketing claim structurally cannot: it is telling a reader how much weight the passing results deserve. On 10 February 2026 it also judged Claude Opus 4.6 to fall short of an AI research and development threshold while conceding that such judgments are becoming harder and more subjective, which is a disclosure of method uncertainty rather than a result.

Measurable criteria have a recognizable shape at smaller scale too. Anthropic's evaluation guidance contrasts safe outputs, which cannot fail, against less than 0.1 percent of outputs out of 10,000 trials flagged for toxicity by the content filter, which names a rate, a population, and a detector.^[2] Any of your claims can be rewritten this way. If it cannot, the claim was never about the system.

One further discipline separates a falsifiable claim from a lucky one: state the trial count and the metric before results are read. pass@k measures the chance of at least one success in k attempts and rises with k, while pass^k measures the chance that all k trials succeed and falls with k, and the two are the same number at k equal to 1.^[3] A threshold that does not say which metric it refers to can be satisfied by choosing the metric after the fact.