How do you measure the retrieval part of a RAG system separately from the generation part?
What the interviewer wants to hear
- Build a labelled set of queries with known relevant chunks
- Use recall@k to check the right chunks are retrieved
- Use MRR or nDCG to check ranking quality
- Evaluate generation separately for faithfulness and correctness
- Track metrics per change: chunking, embeddings, hybrid, reranking
Model answer
I split the evaluation in two. For retrieval I build a labelled set of real queries, each mapped to the chunks or documents that contain the answer, often with help from domain experts or by generating questions from known passages and reviewing them. Then I compute recall@k, because the model cannot use a chunk that was never retrieved, and MRR or nDCG to see whether good chunks rank near the top. Precision@k tells me how much noise goes into the prompt. For generation I separately check faithfulness to the retrieved context and answer correctness, using rubric-based judges calibrated on human labels. Keeping them apart tells me where to fix things: if recall is low I work on chunking, embeddings, hybrid search or reranking; if recall is fine I work on the prompt or model.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium AI Engineer (LLM & Generative AI) questions
- How do you decide between better prompting, RAG and fine-tuning for a new use case?
- How do you evaluate an LLM-powered feature before and after launch?
- Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
- You need an LLM to extract fields from invoices into your database. How do you make this reliable?
- How do you design good tools (functions) for an LLM to call?
- How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
- What kinds of caching can you use in an LLM application, and what are the risks of each?
- What would you put in place to observe and monitor an LLM application in production?
- How would you build and maintain a golden evaluation set for an LLM feature?