hard20 pts · first answer countsevaluationstatistics

Is a 4-point gain real

A new prompt scores 84% on a 100-example eval set versus 80% for the old prompt. A manager wants to ship it as a clear win. What is the right response?

A.Ship it, since 4 points is a 5% relative improvement in accuracyB.Reject it, since any change under 10 points is always pure noiseC.It may be noise: each score has a 95% interval of about ±8 pointsD.Rerun both prompts once more and keep whichever is higher

← Computing nDCG at 3 When a tool call fails →