medium10 pts · first answer countstransformersinferencetokenization

Padding side for batched generation

You batch prompts of different lengths for a decoder-only model and call generate(). Outputs for the shorter prompts are poor. What is the usual fix?

A.Pad on the right so the prompts start at the same positionB.Pad on the left so every prompt ends at the last position before new tokensC.Remove the attention mask so the model sees the paddingD.Use a batch size of exactly the longest prompt length

Discussion 0

Join free to post in the discussion.

Padding side for batched generation

Discussion 0

Related questions