How do tokens affect the cost and speed of an LLM application?

What the interviewer wants to hear

• Pricing is usually per input and output token
• Output tokens are slower and usually cost more
• Long prompts and history increase cost and latency
• Trim context, cache, and choose the right model size

Model answer

Most providers charge per token, with separate prices for input and output tokens, and output is usually more expensive. Speed also depends on tokens: the model processes the input first, which affects time to first token, then generates output one token at a time, so long answers take longer. In a chat app, history grows every turn, so cost per message grows unless I trim or summarise old messages. RAG adds retrieved chunks, so sending ten chunks when three are enough wastes money. To control this, I keep system prompts tight, retrieve fewer but better chunks, ask for concise answers or set max tokens, use prompt caching for repeated prefixes, and send simple tasks to smaller, cheaper models. I log token usage per feature so I can see where money goes.

More easy AI Engineer (LLM & Generative AI) questions

• Explain retrieval-augmented generation (RAG) to a new team member. Why do we use it?
• What are embeddings, and what can you build with them besides RAG?
• Why do LLMs hallucinate, and what simple steps reduce it in an application?
• What do temperature and top-p control, and how would you set them for different tasks?
• What is an LLM agent, and how is it different from a single LLM call?
• How does vector search work, and why is it useful for an AI application?
• Your app starts getting rate-limit errors from the LLM API. How do you handle them?
• What is the difference between zero-shot and few-shot prompting, and when would you use each?
• How do you protect personal data when building a feature on a hosted LLM API?