medium10 pts · first answer countstransformerspositional-encoding

Limit of learned position embeddings

A BERT-style model uses learned absolute position embeddings for 512 positions. What happens if you feed it a 1,000-token input?

A.It reuses position 0 for every token after 512 with no quality changeB.It automatically switches to sinusoidal encodings for the extra tokensC.There are no trained embeddings past 512, so it fails or must truncateD.It processes the input fine because attention is length-independent

Discussion 0

Join free to post in the discussion.

Limit of learned position embeddings

Discussion 0

Related questions