When would you fine-tune a smaller open-weight model instead of using a large hosted model, and how would you do it?
What the interviewer wants to hear
- Narrow, high-volume task with clear success criteria
- Cost, latency, privacy or on-premise requirements
- Generate or collect high-quality training data, possibly distil from a large model
- Parameter-efficient methods like LoRA
- Compare on held-out evals; plan serving and maintenance
Model answer
I consider it when the task is narrow and high volume, such as classifying tickets or extracting fields, and a large model already does it well but is too expensive or slow at scale. Data privacy or on-premise rules are another strong reason. The steps: define the task and an eval set first. Collect training data from labelled history or by having the large model produce outputs that humans spot-check, which is a form of distillation; check the provider terms allow that. Clean the data carefully, since quality matters more than size. Fine-tune with a parameter-efficient method like LoRA, which is cheap and keeps the base model frozen. Compare the fine-tuned model against the large model on held-out data, including edge cases. Finally plan serving with something like vLLM, monitoring, and a process to retrain when data drifts.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard AI Engineer (LLM & Generative AI) questions
- Design a RAG assistant over 2 million internal documents with different access permissions per team.
- How would you protect an LLM application that reads emails and can take actions, against prompt injection and data leaks?
- An agent that completes multi-step tasks works in demos but fails often in production. How do you make it reliable?
- Your LLM feature costs too much and p95 latency is 9 seconds. Walk through how you would reduce both without hurting quality.
- You must self-host an open-weights LLM for a high-traffic product. How do you plan memory, throughput and latency?
- Users say your AI assistant got worse this week, but nothing obvious changed. How do you investigate?
- How do you design an LLM-as-judge evaluation you can actually trust?
- When would you choose a multi-agent architecture over a single agent or a fixed workflow?
- Your agent will connect to several MCP servers, some built by third parties. How do you make this secure?