medium10 pts · first answer countstransformersinference

Reusing a long shared system prompt

Every request to your bot starts with the same 6,000-token system prompt. Which feature cuts both latency and cost most directly?

A.Raising the temperature so the model reads fasterB.Switching from top-p sampling to greedy decodingC.Prefix caching, so the shared prompt's KV cache is reusedD.Increasing max output tokens to 8,000

Discussion 0

Join free to post in the discussion.

Reusing a long shared system prompt

Discussion 0

Related questions