hard20 pts · first answer countstransformerspositional-encoding

Why RoPE encodes relative position

Rotary position embeddings (RoPE) rotate query and key vectors by angles that depend on position. Why does this give relative position information?

A.It adds a learned vector for each absolute positionB.It masks out tokens that are far apartC.It sorts tokens by position before attentionD.The q.k dot product then depends only on the position offset

Discussion 0

Join free to post in the discussion.

Why RoPE encodes relative position

Discussion 0

Related questions