What are tokens and context windows, and why do they matter when you design prompts?

What the interviewer wants to hear

• Tokens are the text pieces models process
• Context window is the total token limit for input and output
• Cost and latency scale with tokens
• Long inputs need trimming, summarising or retrieval
• Token counts differ across languages

Model answer

Tokens are the small chunks of text a model reads and writes, often a word or part of a word. The context window is the maximum number of tokens one request can hold, and that includes the system prompt, conversation history, any documents and the reply. They matter in three ways. First, limits: if I stuff in a huge document and long history, there may be no room for the answer, or older messages get cut. Second, cost and speed: providers charge per token, and output tokens take time to generate, so long prompts and long answers add up across thousands of requests. Third, quality: very long contexts can make models miss details. So I keep prompts focused, trim or summarise history, retrieve only relevant passages, and measure token counts, especially for non-English languages, which often use more tokens.

More easy Prompt Engineer questions

• What makes a good prompt? Walk me through how you would write one for a new task.
• When would you use zero-shot prompting and when would you add few-shot examples?
• How can you use prompting to reduce hallucinations?
• Your product team needs model output that software can read reliably. How do you design the prompt?
• Describe your process for improving a prompt that is giving mixed results.
• Explain temperature and top-p. How would you set them for different tasks?
• Explain retrieval-augmented generation (RAG) to a non-technical colleague. Why would we use it?
• What is chain-of-thought prompting, and when would you use it or avoid it?
• A prompt works on your demo examples but fails for some users. How do you find out why?