Your company is moving its prompts from one model provider to another. How do you plan and run the migration?
What the interviewer wants to hear
- Inventory prompts and their usage, risk and volume
- Run existing eval sets on the new model
- Adapt prompts to new model behaviour and features
- Compare quality, cost, latency and safety
- Gradual rollout with monitoring and rollback
Model answer
I would first list every prompt in production, with its owner, traffic volume, business risk and whether it has an eval set. High-risk prompts without evals get evals first. Then I run each eval set on the new model with the prompts unchanged, to see the baseline differences. Models differ in how literally they follow instructions, verbosity, formatting habits, refusal behaviour and support for features like structured output or tool calling, so I adapt prompts where needed, often simplifying them. I compare quality by category, cost per request, latency and safety tests including injection and refusal cases. Then I roll out gradually, starting with low-risk features, using a small traffic share and monitoring user feedback and error rates, with an easy switch back. I document provider-specific changes for the future.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Prompt Engineer questions
- What is prompt injection, and how would you reduce the risk for an assistant that reads emails and can draft and send replies?
- Design an LLM-as-judge setup to evaluate a summarisation feature. How do you make it trustworthy?
- You need an assistant that serves users writing in Hindi, Tamil and mixed Hinglish. What prompting and evaluation challenges do you expect?
- A high-volume prompt costs too much. How do you reduce cost while protecting quality?
- You are building an agent that reads emails and web pages and can send messages and update a CRM. How do you protect it from indirect prompt injection?
- How would you evaluate a multi-step agent that completes tasks using tools, rather than a single prompt?
- How do you show with confidence that a new prompt is better than the current one, and not just lucky on your test set?
- Design a layered guardrail system for a public-facing LLM assistant. What goes where, and what are the trade-offs?
- An agent runs long tasks that exceed its context window. How do you manage context so quality stays high?