The Prompting Playbook: Better Prompts Start With Evals

At Code w/ Claude London, Anthropic's Margot van Laar used two practical scenarios to explain prompt engineering: repairing a customer-support prompt after a model change and designing a scheduling agent from scratch. Both scenarios began in the same place: an evaluation suite.^[1][2]

That starting point matters. Without an eval, a team can make a prompt cleaner, longer, stricter, or more persuasive without knowing whether the system became more reliable. A passing anecdote can hide a regression elsewhere. A failed example can send the team toward a prompt rewrite when the real limitation sits in the model, tool layer, or application harness.

Anthropic's evaluation guidance makes the same sequence explicit: define measurable success criteria, build tests that reflect the task, iterate, validate, and then ship.^[3] The useful unit is not the prompt by itself. It is the prompt operating inside a versioned system against a representative set of cases.

Van Laar organized the support example around control cases, known edge cases, and capability-boundary cases. A control case should be unambiguous and consistently pass. Edge cases preserve failures the team has already seen. Boundary cases test when the system should answer, use a tool, hand work to a person, or refuse.^[2]

This is a stronger design than collecting a large pile of convenient prompts. The suite should resemble the decisions and failure modes that matter in production. For a support agent, that may include policy conflicts, account-specific facts, calculations, and escalation. For a coding agent, it may include repository discovery, tool errors, tests, and an unsupported completion claim.

Run important cases more than once. Model outputs vary, and one passing sample does not establish consistent behavior. Compare pass consistency and failure categories across prompt, model, and harness versions rather than treating a single run as the answer.^[6]

The talk draws a critical diagnostic line. A new model may have the capability to complete a task but behave differently under the old prompt. That is a prompting problem. It may also lack the capability or reliable mechanism the task requires. No amount of stronger wording repairs that boundary.^[2]

The support demo makes this concrete. Telling a model that correct proration was critical did not create a dependable calculator. Giving the application a calculation tool changed the system's capability. As van Laar put it, "instructions don't add capability." The same principle applies beyond arithmetic: retrieval needs a retriever, private actions need authorized tools, and strict schemas are better enforced by the harness than by repeated pleas in the prompt.^[4][5]

An eval should identify which layer failed before the team chooses a remedy. Otherwise, prompt patches accumulate around architectural problems and become liabilities during the next model migration.

In the first scenario, the production prompt had characteristics familiar to many teams: multiple contributors, no clear owner, copied webpage text, a false human persona, mixed policy and tone guidance, and defensive instructions left behind for earlier models. Cleaning that material and separating role, policy, data, and tone made the prompt easier to reason about and test.^[2]

Structure is not magic. XML tags, headings, and clear sections help because they expose boundaries that both the model and the engineering team can inspect. An output contract provides the same clarity at the other end. When a downstream system requires valid structured data, Anthropic recommends structured outputs rather than relying solely on prompt language.^[5]

Every defensive instruction should carry its history. Link the change to the case that motivated it, the failure it is meant to prevent, and the versions where it was validated. If the underlying behavior changes, the team can remove the patch deliberately instead of preserving it forever.

The talk uses different graders for different kinds of truth. The customer-support responses need judgment about policy and behavior, so an LLM-based grader can apply a detailed rubric. The scheduling problem has explicit staffing constraints, so a programmatic checker can count violations directly.^[2]

That distinction should shape the evaluation stack. Use deterministic graders for facts that code can verify: schema validity, arithmetic, required fields, required tool use, file changes, test results, or policy flags. Check tool sequence only when the order itself is part of the task requirement. Use calibrated LLM judges for relevance, clarity, tone, and other criteria that require interpretation. Keep expert review for high-consequence cases and for calibrating the graders themselves.^[3][6]

Do not let one score erase the reason a run failed. Preserve criterion-level results, grader evidence, and the trace events behind the decision. A prompt may improve answer quality while increasing latency, tool errors, unsupported claims, or escalation failures. Those tradeoffs belong in the release review.

For the scheduling example, the talk compares a simple prompt, stronger reasoning, more adaptive inference, and a generate-evaluate-repair workflow. The last approach separates drafting, critique, and targeted repair instead of asking one prompt to solve and verify everything at once.^[2] Anthropic describes this evaluator-optimizer pattern as a useful fit when evaluation criteria are clear and iterative feedback produces measurable improvement.^[4]

The architecture creates new evaluation obligations. The evaluator can be wrong. The repair loop can repeat without progress. Additional calls can raise latency and cost. Soft runtime preferences can conflict with hard constraints. A production eval therefore needs to score the whole trace, including draft quality, critique accuracy, repair effectiveness, stop behavior, total usage, and final constraint satisfaction.

This is where prompt evaluation becomes agent evaluation. Anthropic's agent-evals guidance defines the transcript as the complete record of a trial and treats the model and agent harness as one evaluated system.^[6] The release question is no longer whether a response looks good. It is whether the system reaches an acceptable result through a controlled, observable, and repeatable process.

A practical implementation starts with a small, versioned contract. Give every case a task identifier, input fixture, expected behavior, failure class, grader, and required evidence. Give every run a prompt version, model identifier, harness version, inference settings, and trace identifier. Then define the conditions that permit promotion.

The release gate should answer concrete questions. Did every critical boundary case pass? Are known regressions still closed? Did any new failure category appear? Is the result stable across repeated trials? Did quality improve by spending more time or tokens than the product can support? Can a reviewer reproduce the result from the stored artifacts?

That is the Vectory interpretation of the prompting playbook: prompt changes are software changes. Evaluate them against the work users need, preserve the evidence behind each result, and promote only the version whose tradeoffs are understood.