GPUs and distributed training
21 practice items · needed by 2 roles
Easy 1Medium 8Hard 12
Practice questions
- Effective batch size across GPUsmedium
- Why fp16 needs loss scaling but bf16 usually does nothard
- Model does not fit in GPU memoryhard
- What ZeRO stage 3 shardshard
- Activation checkpointing trade-offhard
- What FlashAttention changeshard
- Memory for 7B parameters in fp16easy
- Scheduling a GPU podmedium
- Low GPU use under loadmedium
- KV cache size for one long requestmedium
- Training on spot GPUsmedium
- Speeding up training on 8 GPUsmedium
- Why bf16 is easier than fp16medium
- CPU pods filling GPU nodesmedium
- Why LLM servers use continuous batchinghard
- Same seed, different resultshard
- Where to use tensor parallelismhard
- KV cache with grouped-query attentionhard
- Memory to fully fine-tune 7B with Adamhard
- GPU shows 100% but is underusedhard
- Loss spike after resuming traininghard