Your app starts getting rate-limit errors from the LLM API. How do you handle them?

What the interviewer wants to hear

• Recognise 429 responses and honour Retry-After
• Retry with exponential backoff and jitter, with a cap
• Control concurrency with a queue or token budget
• Reduce tokens per request to fit tokens-per-minute limits
• Monitor usage and request higher limits or add fallbacks

Model answer

Rate-limit errors usually come back as HTTP 429. First I make the client handle them properly: read the Retry-After header if present, otherwise retry with exponential backoff plus random jitter, and stop after a few attempts so users are not left waiting forever. Then I look at why we are hitting the limit. Limits are often on both requests and tokens per minute, so trimming long prompts or capping output length can help as much as slowing down. I put a queue or semaphore in front of the API to control concurrency, and move non-urgent work to batch jobs. Finally I add monitoring on usage against limits, ask the provider for a higher tier if needed, and consider a fallback model for peak times.

More easy AI Engineer (LLM & Generative AI) questions

• Explain retrieval-augmented generation (RAG) to a new team member. Why do we use it?
• What are embeddings, and what can you build with them besides RAG?
• Why do LLMs hallucinate, and what simple steps reduce it in an application?
• How do tokens affect the cost and speed of an LLM application?
• What do temperature and top-p control, and how would you set them for different tasks?
• What is an LLM agent, and how is it different from a single LLM call?
• How does vector search work, and why is it useful for an AI application?
• What is the difference between zero-shot and few-shot prompting, and when would you use each?
• How do you protect personal data when building a feature on a hosted LLM API?