hard20 pts · first answer countsevaluation

Perfect on 20 examples

After 40 rounds of tweaking, a prompt scores 100% on your 20 test examples but performs worse than the old prompt on new production data. What happened?

A.The model was updated and broke everythingB.20 examples is too manyC.The prompt was overfitted to the small test set; keep a separate held-out set and grow the eval set with diverse real casesD.Production data must be wrong

← Instructions hidden in user content Can we trust the LLM judge →