Ship Prompt Changes Like Code With a Small Eval Suite

You change a prompt in a shipped retrieval feature, the demo feels sharper, and a user reports a citation that was not in the source. A small eval suite can say whether the edit helped the task you care about or broke the task you forgot.

A Minimum Viable Evaluation Suite (MVES) is an audit-oriented framework for evaluating LLM applications. For prompt work, it is a version-controlled set of task-specific cases with per-task scores. The suite is the merge gate. It gives a prompt change the same review path as a code change: a diff, a test, and a pass/fail result. If a change improves one task and hurts another, the score movement is the argument.

A prompt edit is a merge request

Subjective quality changes with the demo, the model, the day, and the person reading the output. A demo can flatter the change. A prompt sets the model's requirements: what to include, what to omit, how to handle missing information, and when to stop. A small change can sound more confident and still lose the citation, format, or refusal behavior users rely on. Make the prompt a code artifact: diff it, test it, and block it when the suite says no.

A local ablation evaluated five prompt conditions using Ollama with Llama 3 8B Instruct and Qwen 2.5 7B Instruct, with 30 cases per suite. The experiments used the exact Ollama tags llama3:8b and qwen2.5:7b. The repository's evaluation harness supports reproducible local LLM experiments through Ollama.

Build the smallest suite that can catch the failure

• Name the task and the failure you care about. Write the user job in plain language: extract a field, answer from retrieved text, refuse an unsafe request, keep a machine-readable shape. The result is a task label that a teammate can repeat without asking what you meant.
• Collect cases that represent real user work. Include the easy case, the messy case, and the case that already failed in production. You have a version-controlled file where each case has an input, a task, and an expected property to check.
• Score each case against a task-specific property. For extraction, check the field. For retrieval answers, check whether the claim is supported by the source. The output is a per-task score, not an overall vibe.
• Run the prompt conditions against the same suite. Keep the cases fixed while you change the prompt. You get a table where each condition has a score for each task, so a gain in a column cannot hide a loss in another.
• Gate the merge on per-task score movement. Decide before the run which drop is unacceptable. The rule is simple: if a task score falls below the agreed threshold, the change does not ship.

Generic rules are not free. In the tested local conditions, stronger output-contract prompts improved strict extraction for Llama 3 8B Instruct and Qwen 2.5 7B Instruct, while some generic-rule conditions lowered RAG citation/content-compliance. Qwen 2.5's RAG score fell from 26 out of 30 to 9 out of 30 when generic rules were appended to the user prompt. Every added line is a behavior change. If a rule does not map to a case in the suite, it is a guess. If it maps to a case, run it and let the score decide.

Gate the merge on task scores

Do not ask whether the new prompt is better. Ask whether the suite says the task you own got worse. The merge decision should be boring: the suite ran, the scores moved, and the rule was applied.

Keep the suite next to the prompt. When the prompt changes, the cases and the scoring rule change in the same review. Run it on every prompt change, even a small wording tweak. If the suite is too slow, shrink it until it fits the review process.