How do you evaluate an LLM-powered feature before and after launch?

What the interviewer wants to hear

• Build a test set from real queries and edge cases with expected outcomes
• Mix code-based checks, LLM-as-judge with rubric, and human review
• Evaluate components separately (retrieval, generation)
• Run evals on every change like regression tests
• Online: user feedback, task success, cost and latency monitoring

Model answer

Before launch I build an eval set of real or realistic inputs, including common cases, edge cases and adversarial ones, with expected outcomes or grading criteria. I use the cheapest reliable grader for each check: code checks for format, exact values and JSON validity; an LLM judge with a clear rubric for qualities like helpfulness and faithfulness to sources; and human review on a sample, which I also use to calibrate the judge. For RAG I evaluate retrieval separately with recall at k, then generation for correctness and citation support. I run the suite on every prompt, model or pipeline change and compare per category to catch regressions. After launch I track thumbs up and down, task completion, escalation rate, latency and cost, and regularly add failed production cases back into the eval set.

More medium AI Engineer (LLM & Generative AI) questions

• How do you decide between better prompting, RAG and fine-tuning for a new use case?
• Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
• You need an LLM to extract fields from invoices into your database. How do you make this reliable?
• How do you design good tools (functions) for an LLM to call?
• How do you measure the retrieval part of a RAG system separately from the generation part?
• How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
• What kinds of caching can you use in an LLM application, and what are the risks of each?
• What would you put in place to observe and monitor an LLM application in production?
• How would you build and maintain a golden evaluation set for an LLM feature?