hard20 pts · first answer countsevaluationgolden-sets

A suspicious score jump

After a teammate added "a few great examples" to the prompt, eval accuracy jumped from 82% to 96%. You find the examples were copied from the eval set. What is the problem?

A.Nothing; good examples are good examplesB.The eval set is now too smallC.The eval is contaminated: the model is shown answers to test items, so the score no longer predicts real performanceD.The examples made the prompt too long to cache

← Judge and candidate from one family Is a 2-point gain real? →