How would you set up regression testing for prompts so changes can be shipped safely?

What the interviewer wants to hear

• Versioned prompts in git or a prompt tool
• Representative eval set including past failures
• Automated checks plus calibrated LLM judges
• Run in CI on every change and compare to baseline
• Release gates and staged rollout with monitoring

Model answer

I treat prompts like code. Each prompt lives in version control with its model and settings, so every change is reviewed and can be rolled back. I keep an eval set built from real inputs, covering common cases, edge cases and every bug we have fixed. For scoring I use code checks where possible, such as valid JSON, required fields and exact labels, and a calibrated LLM judge with specific criteria for open-ended quality. In CI, any prompt or model change runs the suite and is compared with the current production baseline, per category, not just overall. If key scores drop beyond a set threshold, the change is blocked. Passing changes roll out gradually, for example to a small share of traffic first, while we monitor production metrics and user feedback.

More medium Prompt Engineer questions

• How would you build an evaluation set for a prompt that writes replies to customer complaints?
• When would you split a task into a chain of prompts instead of one large prompt? What are the trade-offs?
• A marketing team wants all generated content to match their brand voice. How do you achieve consistency?
• How do you know when prompt engineering has reached its limit and another approach, such as RAG or fine-tuning, is needed?
• Write the outline of a system prompt for a banking support assistant. What sections would it include and why?
• Design the prompt and retrieval setup for a customer support bot that answers from a help centre. What choices matter most?
• How do you design tool definitions so an LLM agent uses them correctly?
• Your LLM feature works well but is too slow and expensive. How would you reduce cost and latency without hurting quality?
• How would you handle personal data in an LLM workflow that processes customer emails?