Eval set planner
Plan a mix of routine tasks, real failures and difficult cases. Use incidents from your product to keep the set relevant.
About 10 minutes. Nothing leaves your browser. CC0, full guide on GitHub.
Before you write a case
- What decision does this output feed? Name the user task or decision you want to improve.
- What does a wrong answer cost here? Money, time, trust, or a lawyer. A wrong product description and a wrong summary of someone’s medical history do not deserve the same test set.
Five kinds of cases
Adjust this starting mix for your product. Shares must total 100%. Counts are rounded to whole cases; small sets may leave some categories empty.
| Kind of case | Share | Cases |
|---|---|---|
| Golden pathRoutine tasks users expect to work. Include these to catch regressions in core functionality. | % | 0 |
| Real failuresPulled from production: tickets, complaints, anything a human reviewer overrode. If you only build one bucket properly, build this one. | % | 0 |
| Broken and hostile inputMissing fields, wrong language, an empty string, a PDF pasted into a field labelled “name”, text that tries to give the model new instructions. | % | 0 |
| Genuinely unclearCases where two sensible people would disagree. Do not force a correct answer: you are testing whether the system escalates instead of guessing. | % | 0 |
| Should refuseOut of scope, unsafe, or not enough information. Score the refusal as correct. | % | 0 |
Five rules
- Freeze a holdout you do not look at. A widening gap can signal overfitting or a difference in the cases being tested. Investigate before drawing a conclusion.
- Every case needs a one-line reason to exist. Sets rot by accumulation, not by neglect.
- Test what the user gets, not the step in the middle.
- Build rubrics from real failures, not imagined ones.
- Version the set with the prompt. A score without a set version cannot be compared to anything.
How often to run it
- On every prompt change and every model change.
- Review the disagreements weekly: that is where new cases come from.
- Keep cases that protect important behaviour, even when they keep passing. Remove duplicates or cases that no longer match the product.
Review results by category and error severity. An improved average can hide regressions in the cases that matter most.