MLOps Engineer
GPUs and distributed training questions
155 questions on GPUs and distributed training for mlops engineers, easiest first.
Easy 43
- Memory for 7B parameters in fp16easy
- Why deep learning runs on GPUseasy
- Where GPU tensors liveeasy
- Size of a warpeasy
- What a CUDA kernel iseasy
- What an SM iseasy
- What tensor cores doeasy
- Quick check of GPU stateeasy
- Tensors on different deviceseasy
- Reading a CUDA OOM erroreasy
- First thing to try after OOMeasy
- Memory during evaluationeasy
- pin_memory in the DataLoadereasy
- GPU waits for each batcheasy
- Driver and CUDA version checkeasy
- Training runs on CPU by mistakeeasy
- What data parallelism doeseasy
- DataParallel or DistributedDataParalleleasy
- Purpose of all-reduce in DDPeasy
- Backend for multi-GPU trainingeasy
- Launching a multi-GPU jobeasy
- Rank and world sizeeasy
- Every process uses GPU 0easy
- NVLink versus PCIeeasy
- Network between training nodeseasy
- What mixed precision meanseasy
- What autocast doeseasy
- Largest FP16 valueeasy
- Layout of BF16easy
- Eight copies of the same checkpointeasy
- Each GPU sees the same batcheseasy
- Asking Kubernetes for a GPUeasy
- GPU nodes show zero GPUseasy
- GPU metrics in Prometheuseasy
- Sharing one GPU by time-slicingeasy
- GPU bill keeps risingeasy
- Choosing a container base imageeasy
- Where to write checkpointseasy
- Cost of an 8-GPU node for a dayeasy
- Matching the GPU to the jobeasy
- Securing GPUs for a 3-month runeasy
- Memory on common data-centre GPUseasy
- Running many tiny GPU processeseasy
Medium 63
- Scheduling a GPU podmedium
- Low GPU use under loadmedium
- KV cache size for one long requestmedium
- Training on spot GPUsmedium
- Speeding up training on 8 GPUsmedium
- Why bf16 is easier than fp16medium
- CPU pods filling GPU nodesmedium
- Allocated and reserved memorymedium
- empty_cache does not helpmedium
- Memory grows each training stepmedium
- Timing a GPU operationmedium
- Memory coalescingmedium
- If-else inside a warpmedium
- Purpose of shared memorymedium
- Low occupancy in a kernelmedium
- Why GELU and adds are memory-boundmedium
- What kernel fusion savesmedium
- What torch.compile doesmedium
- torch.backends.cudnn.benchmarkmedium
- Memory format for CNNsmedium
- TF32 on Ampere and newermedium
- Odd layer sizes run slowlymedium
- Finding the slow part of a stepmedium
- Gaps in the GPU timelinemedium
- OOM with plenty of reserved memorymedium
- Logging slows the training loopmedium
- Writing a custom fused kernelmedium
- Same order every epochmedium
- Gradient accumulation with DDPmedium
- How DDP hides communicationmedium
- Traffic in ring all-reducemedium
- Building all-reduce from partsmedium
- What ZeRO stage 1 savesmedium
- From ZeRO stage 1 to stage 2medium
- How FSDP runs a layermedium
- Offloading optimizer state to CPUmedium
- BatchNorm with 2 images per GPUmedium
- Learning rate when adding GPUsmedium
- Gradient clipping with FSDPmedium
- Training hangs with no errormedium
- DDP error about unused parametersmedium
- Strong and weak scalingmedium
- GradScaler skipped a stepmedium
- Why keep FP32 master weightsmedium
- Training in FP8medium
- Ops autocast keeps in FP32medium
- How tensor parallelism splits a layermedium
- Training data on a slow network sharemedium
- Millions of tiny image filesmedium
- Distributed job stuck half-startedmedium
- Same GPUs, slower jobmedium
- GPU nodes take long to appearmedium
- Compute-optimal data sizemedium
- Cost of a pretraining runmedium
- Cheaper GPU or faster GPUmedium
- Memory bandwidth by GPU generationmedium
- What NVSwitch addsmedium
- What GPUDirect RDMA doesmedium
- Training pauses while savingmedium
- Catching GPU faults earlymedium
- Scaling efficiency on 8 GPUsmedium
- Bigger global batch stops helpingmedium
- Slow first CUDA call on bare metalmedium
Hard 49
- Why LLM servers use continuous batchinghard
- Same seed, different resultshard
- Where to use tensor parallelismhard
- KV cache with grouped-query attentionhard
- Memory to fully fine-tune 7B with Adamhard
- GPU shows 100% but is underusedhard
- Loss spike after resuming traininghard
- Arithmetic intensity of c = a + bhard
- Arithmetic intensity of a big matmulhard
- Time for one elementwise passhard
- Time for a 4K matmulhard
- Computing MFU for a training runhard
- Estimating total training timehard
- What grows activation memoryhard
- Xid errors in the kernel loghard
- One GPU is slower than its peershard
- Slow transfers between some GPU pairshard
- Error points to the wrong linehard
- Device-side assert in cross-entropyhard
- GPU use swings between 0 and 100%hard
- OOM late in a long evaluationhard
- Overlapping host-to-device copieshard
- Time to all-reduce 7B gradientshard
- Per-GPU state for 13B with ZeRO-3hard
- Idle time in pipeline parallelismhard
- Why 1F1B beats GPipe on memoryhard
- Training on 128K-token sequenceshard
- Placing TP, PP and DP on a clusterhard
- One slow rank in a 256-GPU jobhard
- Accumulated gradients too largehard
- Pure BF16 training stallshard
- Choosing the FSDP wrapping unithard
- Saving a 70B FSDP checkpointhard
- Keep training when a node dieshard
- Multi-node job runs at PCIe speedhard
- Memory saver with the biggest slowdownhard
- Rings versus trees at large scalehard
- How often to checkpointhard
- Failures in a 1,000-GPU jobhard
- Splitting a fixed compute budgethard
- One node's link runs at half speedhard
- H200 or H100 for long-context servinghard
- Pending row remapping on an A100hard
- Data prep and compute in sequencehard
- Expected loss from spot reclaimshard
- Tensor parallel size beyond 8hard
- Capping GPU power in a hot data centrehard
- Bug appears only on some nodeshard
- Different GPU count, different resulthard