hard20 pts · first answer countsdeep-learningtrainingnormalisation
Gradient accumulation with BatchNorm
You replace batch size 256 with batch 16 and 16 accumulation steps. The loss maths matches, yet results are worse. The model uses BatchNorm. Why?
Discussion 0
Join free to post in the discussion.
Related questions
- When early stopping triggersMachine Learning Engineer
- Optimizer steps in one epochMachine Learning Engineer
- Parameters in nn.Linear(10, 5)Machine Learning Engineer
- Why sigmoid slows deep trainingMachine Learning Engineer
- Range of tanhMachine Learning Engineer
- Cross-entropy loss by handMachine Learning Engineer
- Telling autograd to track a tensorMachine Learning Engineer
- Why wrap inference in no_gradMachine Learning Engineer