hard20 pts · first answer countsllm-as-judgeevaluation

Longer answers keep winning

A new prompt produces answers twice as long, and the LLM judge prefers it 75% of the time. Users, however, say the new answers are padded. What is happening?

A.Users are wrong; the judge is more reliableB.The judge needs a higher temperatureC.The prompt needs more examplesD.Verbosity bias: judges often reward length; add explicit conciseness criteria, compare at similar lengths and check against human ratings

← Real sources, wrong claims Tiny wording, big swing →