Design the prompt and retrieval setup for a customer support bot that answers from a help centre. What choices matter most?
What the interviewer wants to hear
- Chunking with titles, hybrid search and reranking
- Clearly delimited, labelled context with IDs
- Answer only from context with a fallback and handoff
- Citations users can follow
- Separate evals for retrieval and answer quality
Model answer
I would start with retrieval, because answers cannot be better than the context. I chunk articles by section and keep the article title with each chunk, use hybrid keyword and vector search so product names and codes match, and rerank the top candidates down to a few. For follow-up questions I rewrite the query into a standalone one. In the prompt, the system message sets the role, tone and rules, and the retrieved chunks go in labelled tags with IDs. I tell the model to answer only from those documents, cite the IDs, and if the answer is not there, say so and offer a human agent. For evaluation I measure retrieval recall on a question set, and separately grade answers for correctness, faithfulness to sources and tone, then monitor fallback rates in production.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Prompt Engineer questions
- How would you build an evaluation set for a prompt that writes replies to customer complaints?
- When would you split a task into a chain of prompts instead of one large prompt? What are the trade-offs?
- A marketing team wants all generated content to match their brand voice. How do you achieve consistency?
- How do you know when prompt engineering has reached its limit and another approach, such as RAG or fine-tuning, is needed?
- Write the outline of a system prompt for a banking support assistant. What sections would it include and why?
- How do you design tool definitions so an LLM agent uses them correctly?
- Your LLM feature works well but is too slow and expensive. How would you reduce cost and latency without hurting quality?
- How would you handle personal data in an LLM workflow that processes customer emails?
- How would you set up regression testing for prompts so changes can be shipped safely?