Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
What the interviewer wants to hear
- Measure retrieval recall with labelled queries first
- Fix parsing and chunking (structure, overlap, metadata)
- Hybrid search with keywords plus vectors
- Query rewriting or expansion; reranking
- Try a better or domain-suited embedding model
Model answer
First I measure. I collect queries where the answer is known to be in the corpus, label the correct chunks, and compute recall at k. Then I look at failures. Often the problem is parsing: tables, PDFs or scanned pages extracted badly. Next is chunking: I split along headings, add overlap, and prepend document and section titles so chunks have context. If users search for codes, names or exact terms, I add keyword search like BM25 and fuse it with vector results. Query rewriting helps when user questions are short or conversational, for example turning "what about refunds?" into a full question using chat history. Retrieving more candidates and using a cross-encoder reranker improves precision. Finally, I test a different embedding model on my labelled set. I change one thing at a time and track recall.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium AI Engineer (LLM & Generative AI) questions
- How do you decide between better prompting, RAG and fine-tuning for a new use case?
- How do you evaluate an LLM-powered feature before and after launch?
- You need an LLM to extract fields from invoices into your database. How do you make this reliable?
- How do you design good tools (functions) for an LLM to call?
- How do you measure the retrieval part of a RAG system separately from the generation part?
- How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
- What kinds of caching can you use in an LLM application, and what are the risks of each?
- What would you put in place to observe and monitor an LLM application in production?
- How would you build and maintain a golden evaluation set for an LLM feature?