hard20 pts · first answer countsmodel-servingdeterminism

Same prompt, temperature 0, different text

With temperature 0, the same prompt gives slightly different answers depending on server load. What is the most likely cause?

A.Temperature 0 still samples at randomB.Batch size changes the float rounding in kernelsC.The weights change between requestsD.The tokenizer becomes random under heavy load

Discussion 0

Join free to post in the discussion.

Same prompt, temperature 0, different text

Discussion 0

Related questions