hard20 pts · first answer countsllm-evaluationstatistics

Twenty variants, one "winner"

You test 20 prompt variants against a baseline, each with a significance test at p < 0.05. One variant shows a significant gain. What is the concern?

A.With 20 tests, about one false "win" is expected by chanceB.p < 0.05 means the gain is certain to be realC.Significance tests cannot be used for promptsD.You should have used p < 0.5 instead

Discussion 0

Join free to post in the discussion.

Twenty variants, one "winner"

Discussion 0

Related questions