hard20 pts · first answer countstransformersinference

Ordering prompts for cache hits

Your prompt is: current date and user name, then a 5,000-token policy manual, then the question. Prompt caching is on, but cache hits are almost zero. What fix gives the most benefit?

A.Shorten the manual to under 1,000 tokens so it fits the cacheB.Move the fixed manual to the very start and put the date and name after itC.Turn off streaming so the cache can store the full outputD.Send the manual as a separate request before every question

Discussion 0

Join free to post in the discussion.

Ordering prompts for cache hits

Discussion 0

Related questions