How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
What the interviewer wants to hear
- Use server-sent events or a streaming HTTP response from the backend
- Keep the API key on the server and proxy the stream
- Handle partial tool-call arguments by buffering until complete
- Handle client disconnects and cancel upstream work
- Validate or moderate content that is shown before it is complete
Model answer
The browser calls my backend, never the provider directly, so the key stays secret. The backend opens a streaming request to the model and forwards text deltas to the client, usually as server-sent events, and the UI appends them as they arrive. This mainly cuts time to first token, which is what users feel. There are several details to handle. Tool-call arguments stream as JSON fragments, so I buffer them per call and parse only when complete. If the user closes the tab, I cancel the upstream request to stop paying for tokens. Errors can happen mid-stream, so the UI needs a clear failure state and retry. Since text is shown before it is finished, any structured output must be validated at the end, and for sensitive apps I run moderation in chunks.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium AI Engineer (LLM & Generative AI) questions
- How do you decide between better prompting, RAG and fine-tuning for a new use case?
- How do you evaluate an LLM-powered feature before and after launch?
- Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
- You need an LLM to extract fields from invoices into your database. How do you make this reliable?
- How do you design good tools (functions) for an LLM to call?
- How do you measure the retrieval part of a RAG system separately from the generation part?
- What kinds of caching can you use in an LLM application, and what are the risks of each?
- What would you put in place to observe and monitor an LLM application in production?
- How would you build and maintain a golden evaluation set for an LLM feature?