A prompt works on your demo examples but fails for some users. How do you find out why?

What the interviewer wants to hear

• Collect real failing inputs and outputs
• Group failures into patterns
• Check the prompt for missing or conflicting instructions
• Fix one cause at a time and re-test on the full set
• Add the failures to the test set

Model answer

First I gather real failing cases from logs or user reports, with the full input, prompt version and output, because demo examples are usually too clean. Then I read them and group them into patterns, for example long inputs, a certain language, missing data or a question type the prompt never mentions. For each pattern I look for the cause in the prompt: an instruction that is missing, vague or contradicted elsewhere, examples that do not cover the case, or a format that breaks. I change one thing at a time and re-run not just the failing cases but the whole test set, so I do not break something else. Finally I add the failing cases to my eval set so the same problem is caught automatically in future.

More easy Prompt Engineer questions

• What makes a good prompt? Walk me through how you would write one for a new task.
• When would you use zero-shot prompting and when would you add few-shot examples?
• How can you use prompting to reduce hallucinations?
• Your product team needs model output that software can read reliably. How do you design the prompt?
• Describe your process for improving a prompt that is giving mixed results.
• Explain temperature and top-p. How would you set them for different tasks?
• Explain retrieval-augmented generation (RAG) to a non-technical colleague. Why would we use it?
• What are tokens and context windows, and why do they matter when you design prompts?
• What is chain-of-thought prompting, and when would you use it or avoid it?