What do temperature and top-p control, and how would you set them for different tasks?

What the interviewer wants to hear

• Both control randomness of token sampling
• Temperature reshapes the probability distribution
• Top-p samples from the smallest set covering probability p
• Low for extraction and code; higher for creative writing

Model answer

Both settings control how random the output is. At each step the model gives a probability for every possible next token. Temperature rescales these probabilities: low temperature makes the top token much more likely, so output is focused and repeatable; high temperature flattens the distribution, so rarer tokens get picked and output is more varied. Top-p, or nucleus sampling, only samples from the smallest set of tokens whose probabilities add up to p, cutting off the unlikely tail. I usually change one, not both. For extraction, classification, SQL or code, I use low temperature for consistency. For brainstorming, marketing copy or story writing, I use a higher value for variety. Even at zero temperature, outputs may not be perfectly identical, so I still design evals and validation instead of relying on determinism.

More easy AI Engineer (LLM & Generative AI) questions

• Explain retrieval-augmented generation (RAG) to a new team member. Why do we use it?
• What are embeddings, and what can you build with them besides RAG?
• Why do LLMs hallucinate, and what simple steps reduce it in an application?
• How do tokens affect the cost and speed of an LLM application?
• What is an LLM agent, and how is it different from a single LLM call?
• How does vector search work, and why is it useful for an AI application?
• Your app starts getting rate-limit errors from the LLM API. How do you handle them?
• What is the difference between zero-shot and few-shot prompting, and when would you use each?
• How do you protect personal data when building a feature on a hosted LLM API?