hard20 pts · first answer countsgpu-computingparallelismlong-context

Training on 128K-token sequences

Even with FlashAttention and activation checkpointing, a single 128K-token sequence does not fit on one GPU. Which parallelism splits the sequence itself across GPUs?

A.Data parallelism with a larger batchB.Pipeline parallelism with more stagesC.Context parallelism, such as ring attentionD.ZeRO stage 1 optimizer sharding

Discussion 0

Join free to post in the discussion.

Training on 128K-token sequences

Discussion 0

Related questions