medium10 pts · first answer countsregression-testingevaluation

A small prompt fix ships

A teammate fixes one failing case by adding a sentence to the prompt and wants to deploy now. What should happen first?

A.Deploy and wait for user complaintsB.Test only the one failing case againC.Run the full eval set and compare scores with the current version to catch regressions elsewhereD.Ask the model whether the change is good

← The judge prefers answer A Which metric for auto-refunds →