medium10 pts · first answer countsllm-as-judgeevaluation

Judge prefers longer answers

Your LLM-as-judge scores long answers higher, even when a short answer is just as correct. What is the best way to reduce this bias?

A.Truncate every answer to 50 words before the judge sees itB.Raise the judge temperature so its scores spread out more evenlyC.Use a rubric that ignores length and calibrate the judge on human labelsD.Switch to a smaller judge model that writes shorter answers itself

← Reciprocal rank fusion scores Dot product on unit vectors →