A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
What the interviewer wants to hear
- Start with prompting and a strong base model as the baseline
- RAG adds knowledge that changes often and allows citations
- Fine-tuning changes behaviour, style or format, not fresh facts
- Consider cost, latency, data available and maintenance
- Decide using an evaluation set, not intuition
Model answer
I start with the cheapest option: a good model with a clear prompt and a few examples, measured on an evaluation set of real questions. If the failures are about missing or out-of-date knowledge, which is likely for internal products, I add RAG. It keeps facts in documents that can be updated daily, supports citations and respects permissions. Fine-tuning is better when the problem is behaviour: a strict output format, a domain style, or a narrow task where a smaller fine-tuned model could replace a big one to cut latency and cost. I would use parameter-efficient methods like LoRA and need a few thousand quality examples. Fine-tuning is a poor way to inject facts that change often. Often the answer is RAG plus a little fine-tuning, chosen by comparing options on the same evaluation set.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Machine Learning Engineer questions
- You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
- What problems does a feature store solve, and when is it worth adding one?
- How do you decide between batch predictions and real-time (online) inference for a model?
- Your neural network loss is not decreasing. How do you debug it?
- After deploying a model, what would you monitor, and how do you handle data drift?
- Walk me through how you would build a RAG system to answer questions over a company knowledge base.
- How would you evaluate an LLM-powered feature before and after launch?
- How do you approach hyperparameter tuning for a model on a limited compute budget?
- Your new model scores far higher than anything before it. How do you check for data leakage?