Why do LLMs hallucinate, and what simple steps reduce it in an application?

What the interviewer wants to hear

• Models predict likely text, not verified facts
• Ground answers with retrieved sources
• Instruct to say "I do not know" and to cite
• Lower temperature for factual tasks; validate outputs

Model answer

LLMs are trained to predict the next likely token, so they produce fluent text even when they do not have the facts. If the answer is not in their training data or context, they may fill the gap with something that sounds right. In an application, the biggest fix is grounding: give the model the relevant facts through retrieval or tool calls, and tell it to answer only from those sources and cite them. I also explicitly allow "I do not know" so the model does not feel forced to guess. For factual tasks I use low temperature. Then I check outputs: for example verifying that cited passages actually contain the claim, validating numbers against the database, and using structured output so fields can be checked. Finally I measure hallucination rate on an eval set.

More easy AI Engineer (LLM & Generative AI) questions

• Explain retrieval-augmented generation (RAG) to a new team member. Why do we use it?
• What are embeddings, and what can you build with them besides RAG?
• How do tokens affect the cost and speed of an LLM application?
• What do temperature and top-p control, and how would you set them for different tasks?
• What is an LLM agent, and how is it different from a single LLM call?
• How does vector search work, and why is it useful for an AI application?
• Your app starts getting rate-limit errors from the LLM API. How do you handle them?
• What is the difference between zero-shot and few-shot prompting, and when would you use each?
• How do you protect personal data when building a feature on a hosted LLM API?