Score agent evals so guardrail violations don't hide
Zero guardrail-violating tasks, split completion from clean completion, and put cost per task on the same card before you choose a model.
| By | Prompt & Product — Newsroom |
|---|---|
| Filed | 20 September 2026 |
| Read | 3 MIN |
If your eval dashboard shows a passing average while a guardrail fired, the average is hiding a launch risk. The fix is a three-number eval card: Completion, Clean Completion, and Cost per Task, with any guardrail violation zeroing the task score. For workflow automation, that card makes the rule visible before the model choice becomes a stakeholder argument.
Zero the task before you average it
Write the guardrail list before you run the suite. Done looks like a short rulebook: no fabricated customer data, no unauthorized payment, no leaked prompt, no off-brand promise. The list should be specific enough that two engineers would mark the same run the same way. A vague guardrail turns the score into a debate about interpretation.
When a guardrail fires, zero the task. The finished run shows a failed-task mark, not partial credit. In AutomationBench-AA, a task receives a zero score when the agent violates a guardrail. Do not let a partial objective score survive the violation. A task that completes the workflow but breaks a rule should be marked as a failed task for the card.
A mean can look healthy while the product is doing something it should not do. Averaging violations into the total gives the model credit for behavior that would fail a launch review.
Report completion and clean completion as separate lines
The finished card needs two quality lines. Completion can count every workflow that reaches its objectives. Clean completion counts only the workflows that reach every objective without a guardrail violation. Completion asks whether the agent can do the job. Clean completion asks whether it can do the job in a way you can ship.
The split is already visible in a model ranking. The v4.3 Intelligence Index swapped out τ³-Banking for AutomationBench-AA and moved Terminal-Bench from version 2.1 to version 4.0. The AutomationBench-AA evaluation runs a held-out set of 657 tasks with Zapier, using benchmark version 1.0.6. In AutomationBench-AA, Score averages task scores over the full 657-workflow set, while Tasks Completed counts only workflows in which all objectives were met with no guardrail violation.
The v4.3 Intelligence Index gave evaluations with private test sets a 45% share of the composite, an increase from 40%. Use that weight when your users are in a regulated or high-risk workflow.
Put cost per task on the same card as quality
Add cost per task to the same card. A model swap decision needs spend before it becomes a quality call. When models tie on the composite index, the cheaper option can be the safer business choice. Cost per task also exposes when a stronger model is paying for a capability your workflow does not use.
In the v4.3 Intelligence Index, GPT-6 Astra (max) and Claude Fable 5.1 (max with fallback) each received an Intelligence Index score of 53, but Astra's average cost per Intelligence Index task was $3.26 versus $7.63 for Fable, making Astra 57% cheaper per task. That gap is the kind of receipt a stakeholder can use. It turns a model comparison into a unit-economics question.
The numbers make the split concrete. In AutomationBench-AA, GPT-6 Astra (max) had a 68.5% Score and a 41.6% clean completion rate, versus 32.1% for Claude Fable 5.1 (max with fallback) and 28.3% for Claude Opus 5 (max). Keep the card small. A designer, a finance partner, and a safety reviewer can all read it. That is the number that tells you whether the workflow can survive a real customer, not just a benchmark run.
Make the card part of the launch review
Put the card in the launch review before the model swap decision. The launch table has columns for completion, clean completion, and cost per task. It should sit next to the user story, not behind a benchmark appendix. A guardrail violation in the clean completion line changes the launch question from whether the agent is clever to whether the agent is allowed to ship.
Use the card to compare options, not to decorate a slide. When an option has higher completion but lower clean completion, say what the user would see. When an option is cheaper but trips the guardrail more often, say what the business would own. Keep the card next to the launch decision until the model is chosen. The card should survive the meeting, not just the deck.
