hard20 pts · first answer countsdeep-learningscaling-lawsmaths
Compute to train a 1B-parameter model
Training compute for a dense transformer is roughly 6 × N × D FLOPs. About how many FLOPs does training a 1-billion-parameter model on 20 billion tokens take?
Discussion 0
Join free to post in the discussion.
Related questions
- Parameters in nn.Linear(10, 5)Machine Learning Engineer
- Why sigmoid slows deep trainingMachine Learning Engineer
- Range of tanhMachine Learning Engineer
- Cross-entropy loss by handMachine Learning Engineer
- Telling autograd to track a tensorMachine Learning Engineer
- Why wrap inference in no_gradMachine Learning Engineer
- Logging loss without leaking memoryMachine Learning Engineer
- Feeding a single image to a CNNMachine Learning Engineer