Your LLM feature works well but is too slow and expensive. How would you reduce cost and latency without hurting quality?

Question

RED SAG · Accepted Answer

I would start by measuring: input and output tokens per request, time to first token, total time and cost per feature. Then I would apply the cheapest safe changes. I would restructure the prompt so the stable parts come first and enable prompt caching. I would trim unused instructions, examples and retrieved chunks, and ask for shorter outputs, since output tokens are slower and pricier. Next I would test routing: send simple requests to a smaller, faster model and keep the large model for hard cases. Work that is not interactive, like nightly labelling, can move to a batch API at a discount. For user-facing replies I would stream so the response feels faster. Every change goes through the eval set first, so I only keep savings that do not lower quality.

Your LLM feature works well but is too slow and expensive. How would you reduce cost and latency without hurting quality?

What the interviewer wants to hear

Model answer

More medium Prompt Engineer questions