How do you show with confidence that a new prompt is better than the current one, and not just lucky on your test set?

What the interviewer wants to hear

• Large enough, representative held-out set
• Paired comparison on the same items
• Repeated runs to estimate variance
• Look at per-slice results and flipped items
• Avoid contamination and overfitting to the eval
• Confirm with online testing or human review

Model answer

First, the test set must be representative and big enough; on 50 items one answer moves the score by 2 points, which is within noise. I keep a held-out set that was never used while tuning the prompt, and I check that no eval items appear as examples in the prompt. I run both prompts on the same items several times and compare paired results: how many items each version fixed and broke, with a simple significance test or confidence interval. I break results down by slice, such as language, topic or input length, because an overall gain can hide a regression for an important group. I also read the flipped items to see if the change is meaningful. Where stakes are high, I confirm with human review or an online A/B test on real traffic.

More hard Prompt Engineer questions

• What is prompt injection, and how would you reduce the risk for an assistant that reads emails and can draft and send replies?
• Design an LLM-as-judge setup to evaluate a summarisation feature. How do you make it trustworthy?
• Your company is moving its prompts from one model provider to another. How do you plan and run the migration?
• You need an assistant that serves users writing in Hindi, Tamil and mixed Hinglish. What prompting and evaluation challenges do you expect?
• A high-volume prompt costs too much. How do you reduce cost while protecting quality?
• You are building an agent that reads emails and web pages and can send messages and update a CRM. How do you protect it from indirect prompt injection?
• How would you evaluate a multi-step agent that completes tasks using tools, rather than a single prompt?
• Design a layered guardrail system for a public-facing LLM assistant. What goes where, and what are the trade-offs?
• An agent runs long tasks that exceed its context window. How do you manage context so quality stays high?