medium10 pts · first answer countstransformerstransformer-architecturetraining

Pre-norm versus post-norm blocks

The original transformer applied LayerNorm after each residual addition (post-norm). Most modern LLMs apply it before each sublayer (pre-norm). Why?

A.Pre-norm removes the need for any learning rate scheduleB.Pre-norm keeps a clean residual path, so deep models train more stablyC.Post-norm cannot be used with a causal attention maskD.Pre-norm halves the parameter count of each block

Discussion 0

Join free to post in the discussion.

Pre-norm versus post-norm blocks

Discussion 0

Related questions