Ship agent evals that catch task failures, safety slips, cost spikes, and latency regressions before users do

A support agent can send a polite reply, leave the ticket unresolved, and still have the user repeat the problem. That is the 'answer-not-resolution' failure: the user gets a reply, not a resolved ticket, not a correct account state, and not a completed task.

The uncomfortable pattern is not just a tooling gap; it is a design failure: teams ship agents faster than they can judge them, and evaluation tooling has not kept pace with the growing diversity of agent frameworks. The fix is a task contract before the agent exists: name the user goal, success criteria, safety constraints, cost budget, latency budget, and feedback signals that tell you the user was helped rather than merely answered. Without that system, every release is a guess dressed up as confidence.

Start with the task contract, not the model

Most agent evals fail because they begin with the model and end with a score. The better move is to begin with the user's task and end with a release decision. The task contract is the first design artifact, and it makes that decision possible.

A task contract turns vague quality into design requirements. If the goal is to resolve a support ticket, success is not a polite reply. It is a resolved ticket, a correct account state, no unauthorized action, and a user who does not need to repeat the problem. If the goal is to draft a document, success includes a usable draft, no private context leak, no fabricated facts, and no budget-burning loop. The contract is the spec that makes the agent accountable.

Instrument the session like a design artifact

Once the contract exists, instrument the agent so every session can be reconstructed as evidence. The session span is not just a log line. It is the record of what the agent saw, what it called, what it changed, and what it returned. If that span does not include message content, tool calls, and user context, the eval is only a rumor. You may know something happened, but you cannot judge whether it was good.

A practical trap is missing message content, because response-quality evaluators error if the span lacks it. The redesign is to treat message content as required evidence, not optional metadata.

Design the spans with the same care as the interface. Record the user's request, the agent's plan, each tool call, the result of each call, the final response, and any user reaction. Record cost and latency at the span level, because a slow tool call can hide inside a fast-looking answer. Record safety-relevant state changes when the agent writes, sends, deletes, or changes permissions.

Choose the harness like a design system

An eval harness is the set of tools that turn sessions into judgments. The same move applies: treat the harness as a design system with reusable components, consistent scoring, and a clear boundary between in-distribution and custom work.

Custom harnesses are still necessary, but they should not become one-off scripts. The judgment is plain: off-the-shelf harnesses are generally better for in-distribution tasks, while custom harnesses keep in-distribution components, such as file editing, aligned with training. The redesign is to use those components as reusable parts of a consistent harness rather than one-off scripts.

The harness should also be framework-agnostic. The design-crit receipts point to the same judgment: the same evaluators can be applied across Strands Agents, LangGraph, OpenAI Agents SDK, LlamaIndex, Google ADK, and Claude Agent SDK after reconstructing sessions from OpenTelemetry spans. The redesign is to make that reconstruction part of the harness, so the same evaluators can judge sessions across frameworks.

Gate releases on the whole product, not one score

The final design move is to make the eval a release gate. A single score is not enough. The gate checks the whole product: task success first, asking whether the agent completed the goal rather than merely sounding plausible; then safety, asking whether it avoided unauthorized actions, privacy leaks, and irreversible mistakes; then cost, asking whether it stayed inside the budget rather than burning money on loops; then latency, asking whether it met the time budget rather than making the user wait; and finally feedback, asking whether users marked the result as helpful, wrong, or unsafe. A release gate is not bureaucracy. It is the design-crit judgment tied to the task contract: do not let a faster agent make a safer-looking mistake, and do not let a cheaper agent quietly fail the user.

Finally, feed production traces into the next eval set. The production-trace flywheel is the design move: traces sharpen the harness, the model, and the context, then the next release gate runs before users meet the change. Back with the support agent, the gate catches the polite unresolved reply by checking ticket resolution, account state, and user feedback, not tone. The verdict is simple: if the task contract is met, ship; if it is not, the gate catches regressions before users do.