hard20 pts · first answer countstransformerspositional-encodingcontext-window

Extending RoPE by position interpolation

A model was trained with RoPE at 4K context. To use it at 16K, a team applies position interpolation before a short fine-tune. What does position interpolation do?

A.Scales positions down by 4 so 16K positions map into the trained 0–4K rangeB.Repeats the 4K position pattern four times across the 16K sequenceC.Replaces RoPE with learned absolute embeddings for positions over 4KD.Drops every fourth token so the sequence fits in 4K positions

Discussion 0

Join free to post in the discussion.

Extending RoPE by position interpolation

Discussion 0

Related questions