Build a CI gate that catches agent tool failures before users do

A merge can pass while the agent is still one broken tool away from a bad state. A seven-agent publishing pipeline delivered over 261 posts. The failures that matter are usually tool failures: timeouts, rate limits, partial writes, and retry loops that look alive but do nothing.

In 2026, a multi-tool agent often makes 3 to 20 tool calls in one task. If each tool call succeeds 98% of the time, a 10-step agent run has at most 81.7% success. That math explains why a happy-path demo is not a reliability proof.

Broken tools are the fixtures

A tool-failure regression harness treats broken tools as first-class test fixtures. You do not only test that the agent calls the right tool. You test what it does when the tool refuses, stalls, returns partial data, or repeats the same failure. Users do not care about your retry count. They care whether the task completes, the state stays consistent, or the stop is clear.

Do not hand-write every trace. Capture a small set from production-like tasks, then prune the noise. A golden trace should be readable by a designer, not only by an engineer. If a teammate cannot say what the agent was trying to do, the trace is too opaque.

Timeouts need a fixture that makes the tool hang long enough for the agent to notice. A rate-limit fixture should return a refusal the agent must respect. Partial writes leave a record in a state that requires reconciliation. Retry storms make the same call fail repeatedly. Stuck loops let the agent repeat a pattern without progress.

AWS describes fault injection as a deliberate way to stress workloads, letting teams watch behavior and strengthen resilience through templates and guardrails like stop conditions. The same idea applies to agent tool calls. A tool is a dependency with failure modes, and those failure modes belong in the test suite.

The gate is a replay loop

• Record golden traces from real tasks. A saved trace for each representative task includes the user request, tool calls, arguments, responses, retries, final answer, and any side effects.
• Write failure fixtures for the tools you actually call. Include timeouts, rate limits, partial writes, retry storms, and stuck loops. For HTTP tools, use RFC 9110 as the reference for core HTTP semantics; Julian Reschke, Roy T. Fielding, Mark Nottingham, and the RFC are associated. Finish when the fixture set covers the failure modes you can actually trigger.
• Replay traces against the fixtures. Run the same task with a tool that fails in a known way. A replay passes when it ends in a recovered state, a degraded state with a clear user-visible reason, or a safe stop.
• Score recovery with a rubric. A finished rubric names the failure and the recovery behavior. Check whether the agent noticed the failure, avoided repeating the same call, preserved partial state, and told the user what it could not do.
• Gate merges on the rubric. The CI job fails on a dropped recovery score, a stuck loop, or an unreconciled partial write.

The common mistake is to test only the final answer. A fluent answer can hide broken tool state. Gate the side effects, not just the prose.

Keep the rubric small enough to argue about in a review. A recovery score should not hide a bad failure. If the agent recovers by destroying the user's state, the score should fail. The rubric is a design decision, not a metric to polish later.

A useful gate starts with harmful tools

Start with the tools that can cause real harm. Any tool that writes, deletes, or publishes deserves a fixture before it ships. A read-only lookup can start with a timeout fixture and a retry fixture. The harness should be steady enough to run on every merge and precise enough to catch the failure that would otherwise reach a user.

If you want cheap local replay, a small local model can keep the loop fast. The author's benchmark data measured Llama 3.1 8B Q4_K_M at a rate of 12 tokens per second on a 16GB M1; the 36GB M4 Max reached 58 tokens per second. That kind of speed matters when the harness runs on every merge.

Keep the rubric visible in the pull request. A short trace diff, a failure name, and a recovery verdict are more useful than a dashboard full of green checks. When the gate is in place, a merge is approved because the agent survived the tool failure it will meet in production.