How would you evaluate an LLM-powered feature before and after launch?

What the interviewer wants to hear

• Define task-specific success criteria and a representative test set
• Use automatic checks: exact match, format validation, groundedness
• Use LLM-as-judge with a rubric, validated against human labels
• Track regressions on every prompt or model change
• Online: user feedback, task success, latency, cost and safety incidents

Model answer

First I define what good looks like for this feature, for example a correct answer, the right JSON format, no made-up facts and a polite tone. I build a test set from real or realistic inputs, including hard and adversarial cases, with reference answers where possible. Some checks are automatic: schema validation, exact match on extracted fields, and whether answers are supported by the retrieved context. For open-ended quality I use an LLM judge with a clear rubric, and I check its agreement with human ratings on a sample and control for position and length bias. This suite runs on every prompt or model change so I catch regressions. After launch I run an A/B test or staged rollout and track task success, thumbs-up rates, escalations, latency, cost per request and safety incidents, and I feed failures back into the test set.

More medium Machine Learning Engineer questions

• You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
• What problems does a feature store solve, and when is it worth adding one?
• How do you decide between batch predictions and real-time (online) inference for a model?
• Your neural network loss is not decreasing. How do you debug it?
• After deploying a model, what would you monitor, and how do you handle data drift?
• Walk me through how you would build a RAG system to answer questions over a company knowledge base.
• How do you approach hyperparameter tuning for a model on a limited compute budget?
• A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
• Your new model scores far higher than anything before it. How do you check for data leakage?