How would you build and maintain a golden evaluation set for an LLM feature?

What the interviewer wants to hear

• Start from real user inputs covering common and edge cases
• Define expected outputs or rubrics with domain experts
• Include known failures and adversarial cases
• Version the set and run it on every prompt or model change
• Refresh it from production as traffic changes

Model answer

I start from real inputs, sampled from logs or pilot users, so the set reflects actual traffic rather than what I imagine. I make sure it covers the main intents plus edge cases: empty input, long input, other languages, and questions we should refuse. For each case I write an expected answer or a grading rubric together with domain experts, since some tasks have many valid answers. Every bug report becomes a new case, so past failures never quietly return. The set is versioned in the repo and runs in CI on every prompt, model or retrieval change, with results compared per category. I keep it large enough to detect the size of change we care about, and refresh it regularly from production so it follows shifts in what users ask.

More medium AI Engineer (LLM & Generative AI) questions

• How do you decide between better prompting, RAG and fine-tuning for a new use case?
• How do you evaluate an LLM-powered feature before and after launch?
• Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
• You need an LLM to extract fields from invoices into your database. How do you make this reliable?
• How do you design good tools (functions) for an LLM to call?
• How do you measure the retrieval part of a RAG system separately from the generation part?
• How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
• What kinds of caching can you use in an LLM application, and what are the risks of each?
• What would you put in place to observe and monitor an LLM application in production?