What would you monitor for an LLM-powered application in production, beyond normal service metrics?
What the interviewer wants to hear
- Traces per request with prompt version, model, retrieval and tool calls
- Token usage and cost per feature or tenant
- Latency split into time to first token and total time
- Quality signals: user feedback, automated evals on samples, refusal and error rates
- Safety signals such as injection attempts and PII in outputs
Model answer
On top of uptime and error rates, I trace every request end to end: prompt template version, model, retrieved documents, tool calls, and the latency of each step. I track token usage and cost per feature and per tenant, because costs can jump when prompts or context sizes change. For latency I separate time to first token from total generation time, since streaming makes the first one most visible to users. Quality is harder, so I combine user feedback such as thumbs up and down, automated evaluations on a sample of production traffic using rubric-based judges checked against human labels, and rates of refusals, empty answers and failed JSON parsing. For safety I monitor guardrail triggers, suspected prompt injection and personal data in outputs. All of this is broken down by prompt and model version, so regressions after a change are easy to spot.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium MLOps Engineer questions
- What does a good CI/CD pipeline for machine learning look like?
- Explain shadow, canary and A/B deployments for models. When do you use each?
- Design an automated retraining pipeline for a model whose data changes weekly.
- What operational challenges come with running a feature store, and how do you handle them?
- How do you run GPU model inference efficiently on Kubernetes?
- How do you design a training pipeline in an orchestrator like Airflow so that it is reliable and easy to rerun?
- How would you estimate the GPUs needed to serve an open-weight LLM for a given number of users?
- How would you define service level objectives for a production model service?
- How do you secure the ML pipeline and model supply chain?