Users say your AI assistant got worse this week, but nothing obvious changed. How do you investigate?

What the interviewer wants to hear

• Confirm with data: feedback, online eval scores, segments
• Check every change: prompts, model or provider version, retrieval index, tools, data
• Compare traces of bad cases to find which step fails
• Check for input drift such as new topics or languages
• Fix, add failing cases to the golden set, and add alerts

Model answer

First I confirm and size the problem with data: feedback rates, retries, online eval scores and abstentions, split by intent, language and customer, to see whether it is broad or one segment. Then I list everything that can change silently: prompt versions, the provider model behind an alias, the embedding model, the index contents, tool APIs, and upstream data. I pull traces of bad examples and walk through each span to find the failing step, for example retrieval returning stale chunks or a tool now returning a new format. I also compare this week inputs with last week to spot drift, such as new topics after a launch. Once I find the cause, I fix it, rerun the golden set, add the failing cases to it, and add alerts or pinned versions so it is caught sooner next time.

More hard AI Engineer (LLM & Generative AI) questions

• Design a RAG assistant over 2 million internal documents with different access permissions per team.
• How would you protect an LLM application that reads emails and can take actions, against prompt injection and data leaks?
• An agent that completes multi-step tasks works in demos but fails often in production. How do you make it reliable?
• Your LLM feature costs too much and p95 latency is 9 seconds. Walk through how you would reduce both without hurting quality.
• When would you fine-tune a smaller open-weight model instead of using a large hosted model, and how would you do it?
• You must self-host an open-weights LLM for a high-traffic product. How do you plan memory, throughput and latency?
• How do you design an LLM-as-judge evaluation you can actually trust?
• When would you choose a multi-agent architecture over a single agent or a fixed workflow?
• Your agent will connect to several MCP servers, some built by third parties. How do you make this secure?