Make agent evals survive the next rule change

Your agent passes the demo. Then the refund policy changes, and the same prompt starts giving confident answers that no longer match the business. A release gate needs checks that the agent understands the business, acts within authority, and keeps learned behavior after new cases and rule changes.

Brackett released the Agent Effectiveness Index (AEI) as a free, open-source benchmark on September 16, 2026. The index gives product designers and engineering leads an audit to run before a production incident.

Map the business before you evaluate the model

Start with the evidence the agent should use. A support agent should cite the current refund window, the exception owner, and the system of record before it recommends a payout. The passing test is a case where the agent refuses to invent a policy and asks for the missing record.

Build a small set of business questions that change when the company changes. Include questions tied to a customer tier, a region, and a recent policy edit. The eval should fail when the agent answers from general training instead of your data. That failure gives the team a concrete reason to stop the release.

Test execution at the limits of authority

A correct answer is not enough if the agent can approve a refund it was never allowed to approve. Put the agent in a case where the right action requires stopping, escalating, or asking for a human decision.

A regression suite should include authority cases: a request that exceeds the agent's spending limit, a conflicting request against a compliance rule, and an ambiguous request that requires clarification. The agent should not solve the ambiguity by guessing. It should preserve the audit trail and hand off cleanly.

Teams often treat a successful demo as proof of authority. If the agent can perform an action outside its granted scope, the eval has failed even when the output looks helpful.

Prove learning survives new cases and rule changes

A prompt update that fixes one ticket does not prove the behavior is durable. The same behavior should hold for a differently phrased case, a returning user, and a revised policy.

Create a learning test that teaches the agent a new rule, runs a fresh case that requires the rule, and reruns the case after a delay or a rule revision. The passing record shows the behavior changed, then stayed changed. If the agent reverts, the eval should flag a persistence failure, not a temporary mistake.

Keep the learning cases separate from the original training examples. The agent should transfer the rule to new cases, not memorize the example. Agent regression testing gives the team a repeatable way to catch regressions during release review.

Turn the audit into a release gate

Make the audit repeatable. Brackett made the complete AEI task set, scoring code, and methodology available on GitHub under the MIT License. Brackett's initial AEI score release compared learning and comprehension for Brackett, OpenAI's Codex, and Anthropic's Claude using one shared demonstration. You can borrow the structure even if your product is not the same as the benchmark: keep the released materials in your own harness, where the team can inspect, version, and rerun them.

The AEI reference method conditions its 10,000 stratified paired block-bootstrap resamples on the declared benchmark population. You do not need to copy that exact method: declare the population you are testing, keep the resampling logic stable, and compare versions against the same declared set.

• Business understanding: the AEI asks whether an agent understands how a particular company actually operates and bases answers on real evidence instead of plausible guesses.
• Operational execution: the benchmark asks whether an agent produces correct outcomes, handles exceptions, and remains within the authority granted to it.
• Learning persistence: the index asks whether an agent's behavior actually changed after instruction and whether that change survives new cases, elapsed time, and rule changes.

Block the launch when any axis fails.