Machine Learning Engineer
Deep learning and PyTorch questions
189 questions on Deep learning and PyTorch for machine learning engineers, easiest first.
Easy 54
- Symptom of a learning rate that is too higheasy
- Shape of a matrix producteasy
- Cosine similarity by handeasy
- Chain rule on a squared termeasy
- ReLU applied to a vectoreasy
- Broadcasting a column and a roweasy
- Dropout after eval()easy
- When early stopping triggerseasy
- Optimizer steps in one epocheasy
- Expected starting loss, 10 classeseasy
- Parameters in nn.Linear(10, 5)easy
- Why sigmoid slows deep trainingeasy
- Range of tanheasy
- Cross-entropy loss by handeasy
- Telling autograd to track a tensoreasy
- Why wrap inference in no_gradeasy
- Logging loss without leaking memoryeasy
- Feeding a single image to a CNNeasy
- Tensors on different deviceseasy
- model(x) versus model.forward(x)easy
- Methods a map-style Dataset needseasy
- Where to set shuffle=Trueeasy
- Default betas in Adameasy
- Saving a trained modeleasy
- Adding weight decay to an optimisereasy
- Size after nn.Flatteneasy
- Padding that keeps image sizeeasy
- Shape after max poolingeasy
- Global average poolingeasy
- Preprocessing for a pretrained ResNeteasy
- What ToTensor does to pixelseasy
- Where random flips belongeasy
- What an RNN carries between stepseasy
- The attention formulaeasy
- Size of each attention headeasy
- How BERT is pretrainedeasy
- Size of an embedding tableeasy
- Patches in a Vision Transformereasy
- Purpose of an autoencoder bottleneckeasy
- Model type for ticket classificationeasy
- Slope of the sigmoid at zeroeasy
- ReLU gradient below zeroeasy
- Chain rule through two linear stepseasy
- Entropy of a fair four-way choiceeasy
- Shape of a cosine LR scheduleeasy
- Every weight set to zeroeasy
- Loss crawls down very slowlyeasy
- Spotting vanishing gradientseasy
- Up-weighting a rare classeasy
- 800 labelled imageseasy
- NaN from a hand-written log losseasy
- Dropout set to 0.9easy
- What DDP syncs between GPUseasy
- DataParallel or DistributedDataParalleleasy
Medium 85
- Different results every inference runmedium
- Early layers stop learningmedium
- Log loss of a poor predictionmedium
- Conv output size with stride 2medium
- Parameters in a conv layermedium
- Gradient of a sum of squaresmedium
- Training loop without zero_gradmedium
- Softmax before CrossEntropyLossmedium
- Shape after a batched matmulmedium
- Why use learning rate warmupmedium
- Initialisation for a deep ReLU networkmedium
- Loss turns NaN in an RNNmedium
- Parameters in a small MLPmedium
- Neurons that always output zeromedium
- Activation in BERT and GPT feed-forwardsmedium
- Softmax by handmedium
- Adding 100 to every logitmedium
- backward() on a vectormedium
- What detach() doesmedium
- torch.cat or torch.stackmedium
- Converting an HWC imagemedium
- Double versus float errormedium
- Forgetting super().__init__()medium
- GPU waits on datamedium
- Where scheduler.step() goesmedium
- BatchNorm statistics at test timemedium
- What LayerNorm normalises overmedium
- Overfit one batch firstmedium
- Where gradient clipping goesmedium
- Why skip connections helpmedium
- Loss for a single-logit classifiermedium
- Target format for CrossEntropyLossmedium
- Two runs give different lossesmedium
- Freezing a pretrained backbonemedium
- torch.inference_modemedium
- Slow first step after torch.compilemedium
- Receptive field of three 3×3 convsmedium
- What a 1×1 convolution doesmedium
- Learning rates when fine-tuningmedium
- Role of the LSTM forget gatemedium
- Output size of a bidirectional LSTMmedium
- Doubling the context lengthmedium
- Why GPT uses a causal maskmedium
- Attention without position informationmedium
- What the KV cache storesmedium
- Where cross-attention gets keysmedium
- How contrastive pretraining worksmedium
- Quantising weights to int8medium
- Active parameters in a mixture of expertsmedium
- Skip connections in U-Netmedium
- Upsampling with a transposed convmedium
- Shifting an image under a conv layermedium
- IoU of two boxesmedium
- Many boxes on one carmedium
- GRU compared with LSTMmedium
- Beam search in decodingmedium
- What skip-gram learns frommedium
- tanh slope from its outputmedium
- Computing log of a sum of exponentialsmedium
- exp of a large logit in float32medium
- Perplexity from cross-entropymedium
- Order of arguments in KL divergencemedium
- Learning rate when batch size growsmedium
- The one-cycle learning rate policymedium
- Reading an LR range testmedium
- Loss jumps at each epoch startmedium
- Training loss higher than validationmedium
- Checking a gradient numericallymedium
- What universal approximation promisesmedium
- Loss suddenly jumps to infmedium
- Setting cudnn.benchmark = Truemedium
- Loss jumps after resuming trainingmedium
- Keeping an exponential moving average of weightsmedium
- Reading a layer's outputmedium
- Gradient noise and batch sizemedium
- Tokens for a compute-optimal modelmedium
- Using argmax inside a lossmedium
- Gradient size of L1 and L2 lossesmedium
- Same data order every epochmedium
- How tensor parallelism splits a layermedium
- DDP error about unused parametersmedium
- Raising GPU throughput when servingmedium
- What PCA components aremedium
- Loss that tolerates a few wild labelsmedium
- Different seeds, same logistic modelmedium
Hard 50
- Why fp16 needs loss scaling but bf16 usually does nothard
- L2 penalty versus AdamWhard
- view() after a transposehard
- Why Adam divides by 1 - beta^thard
- Layers in a plain Python listhard
- Momentum update by handhard
- Scaling inside dropouthard
- Gradient shrink through ten sigmoidshard
- Initialising a tanh networkhard
- Target after label smoothinghard
- Keys prefixed with module.hard
- Error from an in-place operationhard
- Why .grad is None for an intermediatehard
- Calling backward twicehard
- Parameters in a depthwise separable convhard
- An augmentation that changes the labelhard
- Padding in a batch of sentenceshard
- Downside of teacher forcinghard
- Pre-norm transformer blockshard
- Sharing embedding and output weightshard
- The reparameterisation trickhard
- A GAN that draws one facehard
- What a diffusion model learnshard
- Pruning that actually runs fasterhard
- Reach of sliding-window attentionhard
- Area covered by a dilated kernelhard
- Why detectors use focal losshard
- Attention to padding tokenshard
- Triplet loss for one examplehard
- Gradient of softmax cross-entropyhard
- Gradient of a linear layer's weightshard
- BatchNorm on three valueshard
- Long-run step size with momentumhard
- Adam and a scaled-up losshard
- Memory for fp32 Adam traininghard
- Very large batches hurt test accuracyhard
- Validation loss and accuracy both risehard
- Test error falls again with more parametershard
- Where gradient descent gets slowhard
- Fitting completely random labelshard
- Gradient accumulation with BatchNormhard
- Backpropagating through a gradienthard
- Writing a custom autograd Functionhard
- Parameters often excluded from weight decayhard
- Compute to train a 1B-parameter modelhard
- MSE loss that never improveshard
- Idle time in pipeline parallelismhard
- Multi-GPU job that hangs foreverhard
- Margin width of a linear SVMhard
- Promising a delivery time to customershard