What would you put in place to observe and monitor an LLM application in production?
What the interviewer wants to hear
- Traces with spans for each model call, retrieval and tool call
- Record prompt version, model, tokens, cost and latency per span
- Collect user feedback and run online evals on sampled traffic
- Alert on error rates, latency, cost and quality signals
- Handle PII in logs with masking and retention rules
Model answer
I instrument every request as a trace, with spans for each model call, retrieval step and tool call, often using OpenTelemetry-style conventions so it works with existing tooling. Each span records inputs and outputs, prompt version, model, token counts, cost, latency and errors. That lets me debug a single bad answer and also see trends. On top I add product signals such as thumbs up or down, edits and retries, and run automated online evals, like groundedness or format checks, on a sample of traffic. Dashboards and alerts cover error rates, time to first token, p95 latency, cost per request and drops in quality scores, broken down by prompt version so I can tie regressions to deploys. Because traces hold user text, I mask PII and set strict retention and access.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium AI Engineer (LLM & Generative AI) questions
- How do you decide between better prompting, RAG and fine-tuning for a new use case?
- How do you evaluate an LLM-powered feature before and after launch?
- Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
- You need an LLM to extract fields from invoices into your database. How do you make this reliable?
- How do you design good tools (functions) for an LLM to call?
- How do you measure the retrieval part of a RAG system separately from the generation part?
- How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
- What kinds of caching can you use in an LLM application, and what are the risks of each?
- How would you build and maintain a golden evaluation set for an LLM feature?