How would you build an evaluation set for a prompt that writes replies to customer complaints?

What the interviewer wants to hear

• Sample real complaints across types, channels and difficulty
• Include edge cases: angry, vague, multilingual, abusive, out of scope
• Define pass criteria or rubric with domain experts
• Choose grading: human, LLM judge, simple checks
• Keep a held-out portion and update over time

Model answer

I would start with real complaints, with personal data removed, sampled across categories like delivery, refunds and product quality, and across channels and lengths. Then I add edge cases on purpose: very angry customers, vague messages, mixed languages such as Hinglish, abusive text, and requests outside policy. With the support team, I define what a good reply looks like as a rubric: correct policy, empathy, clear next step, right tone, no false promises. Some checks can be automatic, like length or whether a refund amount matches policy; others need an LLM judge calibrated against human scores, plus regular human review. I tag each example by category so I can see where a prompt is weak. I keep part of the set held out to avoid overfitting, and keep adding new real failures.

More medium Prompt Engineer questions

• When would you split a task into a chain of prompts instead of one large prompt? What are the trade-offs?
• A marketing team wants all generated content to match their brand voice. How do you achieve consistency?
• How do you know when prompt engineering has reached its limit and another approach, such as RAG or fine-tuning, is needed?
• Write the outline of a system prompt for a banking support assistant. What sections would it include and why?
• Design the prompt and retrieval setup for a customer support bot that answers from a help centre. What choices matter most?
• How do you design tool definitions so an LLM agent uses them correctly?
• Your LLM feature works well but is too slow and expensive. How would you reduce cost and latency without hurting quality?
• How would you handle personal data in an LLM workflow that processes customer emails?
• How would you set up regression testing for prompts so changes can be shipped safely?