How would you evaluate a multi-step agent that completes tasks using tools, rather than a single prompt?
What the interviewer wants to hear
- Realistic task suite with checkable end states
- Outcome success as the main metric
- Trajectory checks: tool choice, steps, errors, safety
- Multiple trials because runs vary
- Cost, latency and step counts as metrics
- Sandboxed environments and trace review
Model answer
I would build a suite of realistic tasks, each with a starting state in a sandbox and a way to check the end state automatically, such as "the refund exists in the test database with the right amount". Outcome success is the headline metric, because agents can reach correct results by different paths. I would also grade the trajectory: did it choose sensible tools, avoid forbidden actions, recover from errors and ask for confirmation when required. Agent runs vary, so I run each task several times and report success rates and consistency, not single results. I track cost, latency and number of steps, because an agent that succeeds after forty calls may not be usable. Finally, I read traces of failures to find root causes and add new failure cases to the suite.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Prompt Engineer questions
- What is prompt injection, and how would you reduce the risk for an assistant that reads emails and can draft and send replies?
- Design an LLM-as-judge setup to evaluate a summarisation feature. How do you make it trustworthy?
- Your company is moving its prompts from one model provider to another. How do you plan and run the migration?
- You need an assistant that serves users writing in Hindi, Tamil and mixed Hinglish. What prompting and evaluation challenges do you expect?
- A high-volume prompt costs too much. How do you reduce cost while protecting quality?
- You are building an agent that reads emails and web pages and can send messages and update a CRM. How do you protect it from indirect prompt injection?
- How do you show with confidence that a new prompt is better than the current one, and not just lucky on your test set?
- Design a layered guardrail system for a public-facing LLM assistant. What goes where, and what are the trade-offs?
- An agent runs long tasks that exceed its context window. How do you manage context so quality stays high?