hard20 pts · first answer countsgpu-computingreliabilitydistributed-training

One slow rank in a 256-GPU job

In a 256-GPU synchronous training job, per-step time jumps by 30%. Per-rank timings show one rank is consistently slow in its forward pass. What is the effect and response?

A.Only that rank's data is delayed; ignore itB.Other ranks take over its work by themselvesC.The job automatically lowers its batch sizeD.All ranks wait for it; find and replace the faulty node

Discussion 0

Join free to post in the discussion.

One slow rank in a 256-GPU job

Discussion 0

Related questions