hard20 pts · first answer countsllm-as-judgeevaluation

A judge with 90% agreement

In your eval data, 90% of answers are good and 10% are bad. Your LLM judge agrees with human labels 90% of the time. Why is this number alone not reassuring?

A.Agreement above 80% always means the judge is overfittingB.A judge that always says "good" also scores 90% yet catches no bad answersC.Human labels are always less reliable than labels from an LLM judgeD.Agreement only counts when it is measured on the training set instead

← Data leaking through image links Computing nDCG at 3 →