Grade every agent thread without breaking the eval budget

Your eval dashboard is probably showing you a sample, not the product. Stop sampling agent conversations; grade every thread with a specialist judge. When a specialist judge is more accurate and cheaper on your workload, full coverage is the default, not a thin audit sample.

The usual move is to sample production traces and run a frontier LLM-as-judge on the leftovers. That pattern feels responsible. It is usually wrong for trust-critical agents. You are measuring the subset that survived your filter.

Full coverage is a design decision, not a billing accident

Users experience the thread where the agent missed the refund rule, invented a policy, or repeated a refusal. If your evals only see a few of those threads, your dashboard is a mood board, not a trust instrument. The redesign is simple: make the unit of evaluation the thread, not the lucky subset.

A specialist judge can do that job. LangChain launched LangSmith Tuned Evaluators on August 18, 2026. LangChain's first Tuned Evaluator was Perceived Error. That post-trained judge beat every frontier model in its benchmark and cut evaluation cost by 82 percent. Some early-partner workloads reached up to 98 percent savings, with variance tied to thread composition.

Perceived Error is available in the United States for LangSmith Plus and Cloud Enterprise plans. Perceived Error waits until a thread has at least two human-to-AI exchanges and has sat idle for the configured time, then finishes within 12 hours of becoming eligible. LangSmith prices a successful Perceived Error evaluation run at 0.01 LCU.

The cost-coverage rule is one multiplication

Run this rule before you buy, not after the purchase.

• Multiply the per-thread judge cost by your thread volume.
• Compare it with the sampled frontier LLM-as-judge cost for the same coverage gap.
• Require the error taxonomy to match your trust failures, not just generic quality labels.

Do not compare a specialist judge to a frontier model on a single demo thread. Compare them on your production mix, because thread composition changes the bill. Start with the threads that matter most. Some threads are short and safe. Some are long, emotional, and full of edge cases. Your judge should see all of them, but your review queue should prioritize the high-risk ones. Sampling decides what you will never see. Prioritization decides what you will see first.

Keep the raw thread attached to the score. When a judge flags a thread, your team should be able to read the same messages the user sent, not a summarized version.

The strongest objection is that full evaluation will break the budget. It will if you keep paying frontier-model rates for every message. The fix is to stop using the expensive model as the default judge and let a specialist judge carry the production load. The frontier model can still be used for spot checks, hard cases, or building the next judge. It should not be the default cost on every conversation.

The judge has to match the trust failure

A helpfulness score misses the failures you would apologize for: a false promise, a missing handoff, or a privacy leak. The taxonomy has to name those failures. The eval should see missed refund rules in a refund agent, unsafe confidence in medical questions, and unauthorized changes in account actions.

The taxonomy also has to be stable enough to track over time. If the judge can label the same failure in several ways, your team will spend the release cycle arguing about labels instead of fixing the agent. Review the labels with the same care you would give a UI state, because the label is what your team will act on. Keep the categories narrow. Make the examples concrete. Tie each category to a user-visible consequence.

A managed specialist judge is the first purchase, not the last resort. Buy it unless your failure modes are too custom to train or buy. If they are custom, build the eval, and keep the thread as the unit of measurement.