Machine Learning Engineer
GPUs and distributed training questions
20 questions on GPUs and distributed training for machine learning engineers, easiest first.
Easy 2
Medium 6
Hard 12
- Why fp16 needs loss scaling but bf16 usually does nothard
- Model does not fit in GPU memoryhard
- What ZeRO stage 3 shardshard
- Activation checkpointing trade-offhard
- What FlashAttention changeshard
- Idle time in pipeline parallelismhard
- Multi-GPU job that hangs foreverhard
- Base weights for 13B in 4-bithard
- Adam state for a LoRA adapterhard
- NaN loss in fp16 fine-tuninghard
- Checkpointing with frozen inputshard
- Memory for full-vocabulary logitshard