Skip to content
RED SAGSkills
hard20 pts · first answer countsgpu-computingcollectivesperformance

Time to all-reduce 7B gradients

Gradients of a 7B model in BF16 are 14 GB. With ring all-reduce over 8 GPUs at an effective 300 GB/s per GPU, about how long does one all-reduce take, ignoring overlap?

Choose one answer

Discussion 0

Join free to post in the discussion.

Related questions

More GPUs and distributed training questionsMore Performance and debugging questions