How do you design an LLM-as-judge evaluation you can actually trust?

What the interviewer wants to hear

• Use specific rubrics or binary criteria rather than vague 1-10 scores
• Calibrate against human labels and measure agreement per class
• Mitigate position, verbosity and self-preference biases
• Give the judge references or source context where possible
• Re-check calibration when judge model or rubric changes

Model answer

I make the judging task narrow. Instead of "rate quality 1 to 10", I ask specific yes or no questions, such as "is every claim supported by the context?" or "does it follow the refund policy?", each with a clear rubric and examples. I give the judge the reference answer or source documents when I have them. Then I calibrate: experts label a few hundred examples with enough failures in them, and I measure the judge recall on bad answers and a chance-corrected agreement like Cohen kappa, not raw accuracy, which looks good under class imbalance. I counter known biases by swapping order in pairwise tests, penalising padding, and avoiding the same model judging its own outputs. Whenever the judge model or rubric changes, I recalibrate, and I keep sampling cases for human review.

More hard AI Engineer (LLM & Generative AI) questions

• Design a RAG assistant over 2 million internal documents with different access permissions per team.
• How would you protect an LLM application that reads emails and can take actions, against prompt injection and data leaks?
• An agent that completes multi-step tasks works in demos but fails often in production. How do you make it reliable?
• Your LLM feature costs too much and p95 latency is 9 seconds. Walk through how you would reduce both without hurting quality.
• When would you fine-tune a smaller open-weight model instead of using a large hosted model, and how would you do it?
• You must self-host an open-weights LLM for a high-traffic product. How do you plan memory, throughput and latency?
• Users say your AI assistant got worse this week, but nothing obvious changed. How do you investigate?
• When would you choose a multi-agent architecture over a single agent or a fixed workflow?
• Your agent will connect to several MCP servers, some built by third parties. How do you make this secure?