Score model voice before you swap models, so trust doesn't drift

You are about to swap the model behind a user-facing feature. The useful question is whether the voice your users trust stays stable. Qwen3.8 27B answered in about 1,030 words per response, roughly 3.8 times GPT-6 Luna's 270-word average. Score the voice before you ship it.

Measure the default voice before you change it

Pull the default profile first. AA-PersonalityBench tests 30 traits with 30 prompts each, for 900 prompts. For each of the 900 prompts, each model replies once in a plain user turn, with no accompanying system instruction. That gives you the model's native voice, not the voice your product adds on top. Save it as a baseline: one model, one prompt set, one recorded profile, and a note on which traits your users notice most.

Do not paste your production system prompt into the benchmark and call the result your product voice. It measures default communication style; your product voice is the difference you add. Skipping this makes you optimize a demo instead of the behavior your users see.

Your baseline profile should include direct communication metrics, not just trait labels. AA-PersonalityBench measures 21 characteristics from the outputs, including length, structure, punctuation, and emphasis. Those are the variables you can actually tune: a model can be accurate and still feel too long, too formal, or too eager to agree. You are done when the baseline table shows trait scores and communication metrics side by side.

Make taste testable with a weighted spec

Encode the voice spec as a weighted rubric. Give each trait a bounded score, then assign weights that match your product's trust risks. A support assistant may weight clarity and restraint higher than wit; a creative companion may weight warmth higher than brevity. Write what good means before you see the candidate model's output.

Put trust-relevant traits near the top of the list. The transparency index blends five traits, and sycophancy is a negative component, so less sycophancy lifts the score. Easy agreement can look fluent while quietly eroding trust. The artifact is a rubric where the traits users rely on for safety, honesty, or calm have the largest weights.

Weight the traits that appear in real user tasks. Users asking for help need a rubric that tests calm under pressure, not just charm in a happy path. Happy-path examples alone will miss the moments where trust is formed.

Use the confidence band to separate drift from noise

Compare the candidate model to the baseline before you choose. The pairwise judge pool for large language models, which compares outputs two at a time, includes Claude Sonnet 5.5, Gemini 3.8 Flash, and GLM 5.3 Flash. Use the leaderboard's 95% confidence interval as your threshold. A trust-relevant change inside that interval is noise; a change that clears it needs a launch note naming the trait, the direction, and the interval, plus why it matters to the user.

Put the audit in the launch decision

Run the same audit for every model candidate, not just the one you like. Keep the three steps in the launch checklist:

• Pull the 8-index/21-characteristic default profile, the trait indexes plus the output metrics.
• Encode a -2 to +2 weighted voice spec.
• Flag Slop/sycophancy changes, where Slop is the over-agreeable AI tone, only when they clear the ~±26 Elo 95% CI for a typical model, a rating-scale band of about ±5 percentile points.

Save the baseline, the rubric, and the change note in that same place. Helpfulness gains that make the voice more agreeable need a product tradeoff: keep the old model, adjust the prompt, or accept the new voice with a user-facing explanation. Re-run the audit after prompt changes, sampling settings, and model updates, because a voice can drift without a model swap. Pair it with correctness checks and release notes.