How do you know when prompt engineering has reached its limit and another approach, such as RAG or fine-tuning, is needed?

What the interviewer wants to hear

• Measure plateau on eval set despite changes
• Missing knowledge points to retrieval or tools
• Consistent style or format failures at scale may suggest fine-tuning
• Cost or latency of long prompts can justify alternatives
• Some tasks need code or rules instead of models

Model answer

The sign is that scores on the eval set stop improving even after several well-designed changes. Then I look at why it fails. If the model lacks information, such as company policies or current data, no prompt can fix that; retrieval or tool calls are the answer. If the model has the information but keeps failing on a very specific style, format or classification rule, and I have many good examples, fine-tuning may help. If the prompt has grown huge, with dozens of examples, the cost and latency might justify fine-tuning a smaller model. Sometimes the right answer is not an LLM at all: exact calculations, strict validations or simple routing are better done with code. I bring evidence from the eval results when recommending a change.

More medium Prompt Engineer questions

• How would you build an evaluation set for a prompt that writes replies to customer complaints?
• When would you split a task into a chain of prompts instead of one large prompt? What are the trade-offs?
• A marketing team wants all generated content to match their brand voice. How do you achieve consistency?
• Write the outline of a system prompt for a banking support assistant. What sections would it include and why?
• Design the prompt and retrieval setup for a customer support bot that answers from a help centre. What choices matter most?
• How do you design tool definitions so an LLM agent uses them correctly?
• Your LLM feature works well but is too slow and expensive. How would you reduce cost and latency without hurting quality?
• How would you handle personal data in an LLM workflow that processes customer emails?
• How would you set up regression testing for prompts so changes can be shipped safely?