Walk me through how you would build a RAG system to answer questions over a company knowledge base.

What the interviewer wants to hear

• Ingestion: parse, clean and chunk documents with metadata
• Embed chunks and index them; consider hybrid search with BM25
• Retrieve, rerank and fit the best chunks into the prompt
• Prompt the model to answer from context and cite sources
• Evaluate retrieval and answer quality with a labelled test set
• Handle access control, freshness and monitoring

Model answer

I start with ingestion: parse documents, strip boilerplate, and split them into chunks of a few hundred tokens along headings, keeping metadata such as source, date and permissions. I embed each chunk and store it in a vector index, and I add BM25 keyword search because users often search exact product names or codes. At query time I retrieve candidates from both, merge them, and run a cross-encoder reranker to pick the top few that fit the context budget. The prompt tells the model to answer only from the provided passages, cite them, and say when it does not know. Before launch I build an evaluation set of real questions with known source passages, measure retrieval recall at k and answer faithfulness, and iterate on chunking and retrieval first. In production I enforce document permissions at retrieval time, re-index on updates, and monitor no-answer rates and user feedback.

More medium Machine Learning Engineer questions

• You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
• What problems does a feature store solve, and when is it worth adding one?
• How do you decide between batch predictions and real-time (online) inference for a model?
• Your neural network loss is not decreasing. How do you debug it?
• After deploying a model, what would you monitor, and how do you handle data drift?
• How would you evaluate an LLM-powered feature before and after launch?
• How do you approach hyperparameter tuning for a model on a limited compute budget?
• A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
• Your new model scores far higher than anything before it. How do you check for data leakage?