hard20 pts · first answer countsllm-servingllm-security
Semantic cache returns wrong answers
An LLM app added a semantic cache that returns a stored answer when a new question embedding is close to a past one. Users now sometimes get answers to different questions, including another customer account details. What fixes this?
Discussion 0
Join free to post in the discussion.
Related questions
- Memory for 7B parameters in fp16MLOps Engineer
- KV cache size for one long requestMLOps Engineer
- Serving many fine-tuned variantsMLOps Engineer
- Instructions hidden in a documentMLOps Engineer
- Why LLM servers use continuous batchingMLOps Engineer
- Where to use tensor parallelismMLOps Engineer
- KV cache with grouped-query attentionMLOps Engineer
- Memory for 4-bit weightsMachine Learning Engineer