Design a RAG assistant over 2 million internal documents with different access permissions per team.

What the interviewer wants to hear

• Ingestion pipeline: parsing, chunking, metadata, incremental updates
• Hybrid retrieval with reranking
• Permission filtering at retrieval time (document-level ACLs)
• Citations, refusal when unsupported
• Evaluation, monitoring, cost and latency budgets

Model answer

I would start with ingestion: connectors pull documents from each source, parse them carefully (including PDFs and tables), split them along structure, and store chunks with metadata such as source, date, owner and access control list. Updates are incremental using change events, and deletions remove vectors quickly. Retrieval uses hybrid search, vector plus BM25, and always applies a permission filter based on the user identity at query time, so the model never sees chunks the user cannot access; filtering after generation is too late. A reranker narrows to the top few chunks. The generator is instructed to answer only from sources, cite them, and say when it cannot find an answer. I build an eval set per department, measure retrieval recall and answer faithfulness, and monitor latency, cost per query, feedback and stale content.

More hard AI Engineer (LLM & Generative AI) questions

• How would you protect an LLM application that reads emails and can take actions, against prompt injection and data leaks?
• An agent that completes multi-step tasks works in demos but fails often in production. How do you make it reliable?
• Your LLM feature costs too much and p95 latency is 9 seconds. Walk through how you would reduce both without hurting quality.
• When would you fine-tune a smaller open-weight model instead of using a large hosted model, and how would you do it?
• You must self-host an open-weights LLM for a high-traffic product. How do you plan memory, throughput and latency?
• Users say your AI assistant got worse this week, but nothing obvious changed. How do you investigate?
• How do you design an LLM-as-judge evaluation you can actually trust?
• When would you choose a multi-agent architecture over a single agent or a fixed workflow?
• Your agent will connect to several MCP servers, some built by third parties. How do you make this secure?