Your LLM feature costs too much and p95 latency is 9 seconds. Walk through how you would reduce both without hurting quality.

What the interviewer wants to hear

• Measure: tokens, calls per request, time breakdown
• Shorter prompts and context; fewer chained calls
• Model routing: smaller models for easy requests
• Prompt caching, response caching, batching
• Streaming for perceived latency; evals to protect quality

Model answer

I start by measuring where time and money go: input and output tokens per request, number of model calls, retrieval time and time to first token. Often there are quick wins: a bloated system prompt, too many retrieved chunks, or several sequential calls that could run in parallel or be merged. I reduce output length with clear instructions and max tokens, since output tokens are slow and costly. I add model routing, where a small fast model handles simple requests and only hard ones go to the large model; sometimes a fine-tuned small model replaces the large one for a narrow task. Prompt caching cuts cost for repeated prefixes, and caching full responses helps for repeated questions. Streaming improves perceived latency. For offline jobs I use batch APIs. Every change is checked against the eval set so quality does not slip.

More hard AI Engineer (LLM & Generative AI) questions

• Design a RAG assistant over 2 million internal documents with different access permissions per team.
• How would you protect an LLM application that reads emails and can take actions, against prompt injection and data leaks?
• An agent that completes multi-step tasks works in demos but fails often in production. How do you make it reliable?
• When would you fine-tune a smaller open-weight model instead of using a large hosted model, and how would you do it?
• You must self-host an open-weights LLM for a high-traffic product. How do you plan memory, throughput and latency?
• Users say your AI assistant got worse this week, but nothing obvious changed. How do you investigate?
• How do you design an LLM-as-judge evaluation you can actually trust?
• When would you choose a multi-agent architecture over a single agent or a fixed workflow?
• Your agent will connect to several MCP servers, some built by third parties. How do you make this secure?