hard20 pts · first answer countsgpu-computingparallelism
Idle time in pipeline parallelism
A model is split into p = 4 pipeline stages, and each batch is split into m = 12 micro-batches using a GPipe or 1F1B schedule. What fraction of time is the pipeline bubble, using (p − 1) / (m + p − 1)?
Discussion 0
Join free to post in the discussion.