hard20 pts · first answer countsllm-evaluation

LLM judge prefers the first answer

In pairwise evals, your LLM judge picks whichever answer is shown first about 65% of the time, even when you swap the two. How should you make the eval more reliable?

A.Always show your own model's answer firstB.Raise the judge temperature to add varietyC.Judge each pair in both orders and count only consistent winsD.Drop every comparison that ends in a tie

Discussion 0

Join free to post in the discussion.

LLM judge prefers the first answer

Discussion 0

Related questions