hard20 pts · first answer countsevaluationmetrics

Is a 2-point gain real?

Prompt B scores 84% and prompt A 82% on a 50-item eval set. Your manager wants to ship B because it is "better". What is the most accurate response?

A.A 2-point gap on 50 items is one item and may be noise; look at which items changed, use a larger set or repeated runs, and check significanceB.B is clearly better; ship itC.A is better because it was firstD.Evals cannot compare prompts at all

← A suspicious score jump Follow-up questions fail in RAG →