medium10 pts · first answer countsgpu-computingscaling-lawsdistributed-training
Bigger global batch stops helping
Doubling GPUs and global batch size used to halve training time, but beyond a batch of about 4 million tokens you need almost as many steps as before. What is this limit called?
Discussion 0
Join free to post in the discussion.