MLOps Engineer
Resilience and fault tolerance questions
20 questions on Resilience and fault tolerance for mlops engineers, easiest first.
Easy 5
Medium 3
Hard 12
- Retries that make an outage worsehard
- Error budget for 99.9%hard
- Xid errors in the kernel loghard
- One GPU is slower than its peershard
- One slow rank in a 256-GPU jobhard
- Keep training when a node dieshard
- How often to checkpointhard
- Failures in a 1,000-GPU jobhard
- Pending row remapping on an A100hard
- Bug appears only on some nodeshard
- Spot GPUs for an online servicehard
- Overload during a sale eventhard