hard20 pts · first answer countsllm-as-judgeevaluation

Can we trust the LLM judge

You use an LLM judge to score answer helpfulness from 1 to 5. Before relying on it, what is the most important step?

A.Ask the judge to always explain its score in detailB.Use the same model as the one being judgedC.Increase the scale to 1 to 100 for more precisionD.Compare judge scores with human labels on a sample, measure agreement, and refine the rubric until they match well

← Perfect on 20 examples Switching to a newer model →