medium10 pts · first answer countsevaluationllm-as-judge

Pairwise LLM judge bias

You use an LLM as a judge to compare answer A and answer B. You notice it prefers whichever answer is shown first. What is a good mitigation?

A.Run each comparison twice with the order swapped and only count consistent winsB.Always show the new model answer firstC.Raise the judge temperature to 1.5D.Ask the judge to answer in one word only

← Making prompt caching work Cosine similarity by hand →