How do you decide between better prompting, RAG and fine-tuning for a new use case?
What the interviewer wants to hear
- Start with prompting and an eval set as a baseline
- RAG for knowledge: private, large or changing data
- Fine-tuning for behaviour, format, style, or a smaller cheaper model
- Consider data availability, cost, latency and maintenance
- Often combined
Model answer
I start with prompting and build an eval set first, so I can measure every option. Good instructions and a few examples solve many problems cheaply and quickly. If the gap is missing knowledge, such as company documents, product data or anything that changes often, I add RAG or tool calls, because fine-tuning is a poor way to store facts and is hard to update. If the gap is behaviour, like a strict output format, a domain writing style, or a classification task the model keeps getting wrong despite good prompts, fine-tuning helps, especially when I have hundreds or thousands of good examples. Fine-tuning a smaller model can also cut cost and latency once a large model has proven the task. These approaches combine: a fine-tuned model can still use RAG. I always compare against the prompted baseline.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium AI Engineer (LLM & Generative AI) questions
- How do you evaluate an LLM-powered feature before and after launch?
- Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
- You need an LLM to extract fields from invoices into your database. How do you make this reliable?
- How do you design good tools (functions) for an LLM to call?
- How do you measure the retrieval part of a RAG system separately from the generation part?
- How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
- What kinds of caching can you use in an LLM application, and what are the risks of each?
- What would you put in place to observe and monitor an LLM application in production?
- How would you build and maintain a golden evaluation set for an LLM feature?