Design an LLM-as-judge setup to evaluate a summarisation feature. How do you make it trustworthy?
What the interviewer wants to hear
- Clear rubric with specific criteria, ideally pass/fail per criterion
- Provide source text to judge faithfulness
- Calibrate against human labels and measure agreement
- Mitigate biases: length, position, self-preference
- Monitor and re-check judge over time
Model answer
I start by defining what a good summary means with the stakeholders: faithful to the source, covers the key points, right length, clear. Each criterion gets its own judge question, often pass or fail with a short reason, because separate binary checks are more reliable than a single vague 1 to 10 score. The judge always sees the source text so it can check faithfulness, and it is asked to list unsupported claims. Next I calibrate: humans label a few hundred summaries, I compare the judge with them, measure agreement, and refine the rubric and examples until agreement is high. I watch for known biases, such as preferring longer answers, and for pairwise comparisons I swap the order. I use a strong judge model, keep it fixed when comparing versions, and periodically re-check it against fresh human labels.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Prompt Engineer questions
- What is prompt injection, and how would you reduce the risk for an assistant that reads emails and can draft and send replies?
- Your company is moving its prompts from one model provider to another. How do you plan and run the migration?
- You need an assistant that serves users writing in Hindi, Tamil and mixed Hinglish. What prompting and evaluation challenges do you expect?
- A high-volume prompt costs too much. How do you reduce cost while protecting quality?
- You are building an agent that reads emails and web pages and can send messages and update a CRM. How do you protect it from indirect prompt injection?
- How would you evaluate a multi-step agent that completes tasks using tools, rather than a single prompt?
- How do you show with confidence that a new prompt is better than the current one, and not just lucky on your test set?
- Design a layered guardrail system for a public-facing LLM assistant. What goes where, and what are the trade-offs?
- An agent runs long tasks that exceed its context window. How do you manage context so quality stays high?