When would you choose a multi-agent architecture over a single agent or a fixed workflow?

What the interviewer wants to hear

• Prefer fixed workflows when steps are known in advance
• A single agent with good tools covers most open-ended tasks
• Multi-agent helps with parallel subtasks or context isolation
• Multi-agent adds cost, latency, coordination errors and harder debugging
• Decide with evals and traces, not by default

Model answer

I start with the simplest option that works. If the steps are known, such as classify, retrieve, draft, check, a fixed workflow is cheaper, faster and easier to test. If the path depends on what the model finds, a single agent with a well-designed tool set handles most cases. I move to multiple agents only for a clear reason. One is parallelism: an orchestrator can send several subagents to research different parts at once. Another is context isolation: each subagent works in its own fresh context and returns a short summary, which keeps the main context clean on long tasks. The costs are real: many more tokens, more latency, lost information at handoffs, and harder debugging. So I compare designs on the same eval set and use tracing across agents before committing.

More hard AI Engineer (LLM & Generative AI) questions

• Design a RAG assistant over 2 million internal documents with different access permissions per team.
• How would you protect an LLM application that reads emails and can take actions, against prompt injection and data leaks?
• An agent that completes multi-step tasks works in demos but fails often in production. How do you make it reliable?
• Your LLM feature costs too much and p95 latency is 9 seconds. Walk through how you would reduce both without hurting quality.
• When would you fine-tune a smaller open-weight model instead of using a large hosted model, and how would you do it?
• You must self-host an open-weights LLM for a high-traffic product. How do you plan memory, throughput and latency?
• Users say your AI assistant got worse this week, but nothing obvious changed. How do you investigate?
• How do you design an LLM-as-judge evaluation you can actually trust?
• Your agent will connect to several MCP servers, some built by third parties. How do you make this secure?