hard20 pts · first answer countsoptimisationregularisation

L2 penalty versus AdamW

Why is AdamW preferred over adding an L2 penalty to the loss when using Adam?

A.AdamW does not use momentum, so it is more stableB.With Adam, L2 gradients get rescaled by the adaptive learning rate, so the penalty is uneven; AdamW applies weight decay directly to the weights, decoupled from the gradientC.AdamW removes the need for a learning rateD.L2 penalties are not differentiable

Discussion 0

Join free to post in the discussion.

L2 penalty versus AdamW

Discussion 0

Related questions