What kinds of caching can you use in an LLM application, and what are the risks of each?

What the interviewer wants to hear

• Provider prompt caching for repeated prefixes needs stable prompt order
• Exact-match response caching is safe but hits rarely
• Semantic caching raises hit rate but risks false hits
• Cache keys must include user, permissions, model and prompt version
• Use expiry and invalidation when source data changes

Model answer

I think in three layers. First, provider prompt caching: if long system prompts, tool definitions or documents are a stable prefix, the provider reuses that work, cutting cost and latency, so I put static content first and changing content last. Second, exact-match caching of full responses keyed by normalised input, model and prompt version. It is safe but only hits for repeated questions. Third, semantic caching, which returns a stored answer for a similar question. It raises hit rates but can return wrong answers when phrasing is close but meaning differs, so I use a strict threshold, measure false hits on labelled pairs, and skip it for actions. For all of them, the cache key must include user or tenant and permissions so nobody gets someone else data, and entries need expiry when sources change.

More medium AI Engineer (LLM & Generative AI) questions

• How do you decide between better prompting, RAG and fine-tuning for a new use case?
• How do you evaluate an LLM-powered feature before and after launch?
• Your RAG bot often answers "I could not find this" even though the answer is in the documents. How do you improve retrieval?
• You need an LLM to extract fields from invoices into your database. How do you make this reliable?
• How do you design good tools (functions) for an LLM to call?
• How do you measure the retrieval part of a RAG system separately from the generation part?
• How would you implement streaming LLM responses in a web app, and what problems do you need to handle?
• What would you put in place to observe and monitor an LLM application in production?
• How would you build and maintain a golden evaluation set for an LLM feature?