medium10 pts · first answer countstransformersembeddingsattention

Pooling a decoder model into one vector

You want a sentence embedding from a decoder-only LLM's hidden states. Why is taking the first token's hidden state a poor choice?

A.The first token's vector is always all zerosB.The first token is removed by the tokenizerC.The first token is reserved for the output logitsD.With causal attention, the first token has seen only itself

Discussion 0

Join free to post in the discussion.

Pooling a decoder model into one vector

Discussion 0

Related questions