Ship an AI Feature, Not Just a Demo, With Coding Agents

You are staring at a polished agent demo that looks shippable. Sierra's top standalone setup, using Claude Opus 5 with max reasoning in Claude Code, passed 23.9% of held-out customer-service tasks. The same model class reached 82.2% on those tasks with an engineer who had deep context. The gap points to context, and context is a design surface. Grade the finished agent on unseen end-to-end tasks, then audit the build for shallow research, too few client questions, and unhandled constraints.

The demo is the wrong artifact

Coding agents are good at making the happy path look finished. They generate a clean transcript, a plausible answer, and a UI that seems ready. The artifact is the customer-service agent, not the story the build tells about itself.

In the benchmark, a coding agent builds inside an isolated kit, may question a simulated client, and returns a finished, runnable customer-service agent. The finished agent is then scored on unseen customer-service tasks, and the developer's reward equals that agent's pass rate. The arXiv paper, τ^τ-Bench, lists Quan Shi and three other authors. The first arXiv version was submitted on September 4, 2026. The public hyper-τ-bench codebase ships 53 release tasks: six airline_plus, six retail_plus, six telecom, and 35 banking_knowledge.

Score the finished agent before you trust the process

A demo can hide a brittle state machine. A held-out end-to-end task exposes it. The task should be realistic, end to end, and absent from the construction context. It should require the agent to understand the request, call the right tools, handle a failure, and return a useful answer. It also needs a tool call, a state change, and a user-visible outcome.

If your team uses coding agents to assemble AI features, make the finished agent the unit of agent eval. Do not grade the transcript. Do not grade the first working example. Grade the executable artifact on tasks it has not seen.

Build a small eval harness around the artifact. Keep the tasks hidden from the agent during construction. Reuse the same tasks across releases so a new build can be compared against the last one. This turns a one-off demo into a repeatable eval loop.

Audit the build for the failure modes that make demos look shippable

The build trajectory is evidence. A shallow build can still produce a demo. It usually shows the same tells: the agent skipped research, asked no client questions, shipped the first design, and ignored API or cost constraints. A demo can pass because the user is forgiving. A production feature cannot.

The checklist below is the redesign. It turns the audit into a repeatable check. Name the failure mode when you send it back, so the next build fixes the cause, not the symptom. Each item targets a tell and can be checked quickly.

• Run a quick held-out task: pick a customer request the agent did not see during construction and record pass or fail.
• Count the client questions: open the transcript and flag a build with no questions about users, constraints, or edge cases.
• Search for shallow research: look for any source, spec, or prior case beyond the first working example.
• Check the first design: compare the initial implementation with the final one and note if it shipped unchanged.
• Test an API failure: inject a timeout or error and confirm the agent has a fallback.
• Test a cost constraint: set a rate limit or budget cap and confirm the agent degrades gracefully.
• Verify clean startup: start the final agent in a fresh environment and confirm it runs without hand-holding.