hard20 pts · first answer countsevaluationmetrics

Scores change between runs

Running the same prompt on the same eval set gives 79%, 83% and 81% on three runs. How should you report and compare prompt versions?

A.Run several times per version and report the mean with its spread, and compare versions using the same itemsB.Report the best runC.Report only the first runD.Stop using evals because they are random

← Errors flowing down a chain Reasoning model bill surprise →