Skip to content
RED SAGSkills

Deep Learning interview questions and answers

Deep learning questions cover how neural networks learn and what to do when training goes wrong.

Below are 60 Deep Learning questions with the correct answer and a short explanation, grouped from fresher to experienced level. Try each one before you open the answer.

Take the Deep Learning MCQ testPractise 250+ Deep Learning questions for points

What Deep Learning interviews usually check

  • ✓Backpropagation and gradient descent
  • ✓Activations, loss functions and optimisers
  • ✓CNNs, RNNs and attention
  • ✓Vanishing gradients, normalisation and dropout
  • ✓Debugging training runs

Deep Learning interview questions for freshers easy

1. Symptom of a learning rate that is too high

During training, the loss jumps up and down wildly and sometimes becomes NaN. What is the most likely cause?

  1. A.The batch size is too large
  2. B.The learning rate is too high
  3. C.The model has too few parameters
  4. D.The training set is perfectly balanced
Show answer

Answer: B. The learning rate is too high

A learning rate that is too high makes each update overshoot the minimum, so the loss oscillates or explodes, which can lead to NaN values. Lowering the learning rate, using warmup or gradient clipping usually fixes it.

2. Chain rule on a squared term

For f(x) = (3x + 1)^2, what is the derivative f'(x) at x = 1?

  1. A.8
  2. B.12
  3. C.16
  4. D.24
Show answer

Answer: D. 24

By the chain rule, f'(x) = 2(3x + 1) * 3 = 6(3x + 1). At x = 1 this is 6 * 4 = 24. Backpropagation is this same rule applied layer by layer.

3. Parameters in nn.Linear(10, 5)

How many trainable parameters does nn.Linear(10, 5) have?

  1. A.55
  2. B.50
  3. C.15
  4. D.60
Show answer

Answer: A. 55

The weight matrix is 5 × 10 = 50 values, and there is one bias per output, 5 more. Total 55. With bias=False it would be 50.

4. Why sigmoid slows deep training

For large positive or negative inputs, what happens to the gradient of the sigmoid activation?

  1. A.It grows without limit
  2. B.It stays at exactly 1
  3. C.It gets close to zero
  4. D.It flips sign each step
Show answer

Answer: C. It gets close to zero

The sigmoid flattens towards 0 and 1 at the extremes, so its slope σ(x)(1 − σ(x)) approaches zero. Its maximum is only 0.25 at x = 0. Stacking many such layers multiplies small numbers and gradients vanish.

5. Feeding a single image to a CNN

A CNN expects input of shape (N, C, H, W). You have one image tensor of shape (3, 224, 224). Which call makes it valid?

  1. A.img.unsqueeze(0)
  2. B.img.squeeze(0)
  3. C.img.flatten()
  4. D.img.permute(1, 2, 0)
Show answer

Answer: A. img.unsqueeze(0)

unsqueeze(0) inserts a new dimension of size 1 at the front, giving (1, 3, 224, 224), a batch of one. squeeze removes size-1 dimensions, and permute(1, 2, 0) would give channels-last (224, 224, 3).

6. Saving a trained model

What is the recommended way to save a PyTorch model for later use?

  1. A.pickle the whole Python process
  2. B.torch.save(model.forward, path)
  3. C.Print the weights and copy them
  4. D.torch.save(model.state_dict(), path)
Show answer

Answer: D. torch.save(model.state_dict(), path)

state_dict holds only parameter and buffer tensors, so it is portable across code changes and safe to load with weights_only=True. Saving the whole model pickles class references, which breaks when code moves. For resuming training, also save the optimiser and scheduler state_dicts.

7. Shape after max pooling

A tensor of shape (8, 32, 56, 56) goes through nn.MaxPool2d(kernel_size=2, stride=2). What is the output shape?

  1. A.(8, 16, 56, 56)
  2. B.(8, 32, 28, 28)
  3. C.(8, 32, 54, 54)
  4. D.(4, 32, 28, 28)
Show answer

Answer: B. (8, 32, 28, 28)

Pooling with a 2×2 window and stride 2 halves each spatial dimension and leaves the batch and channel dimensions alone. 56 / 2 = 28.

8. What an RNN carries between steps

In a simple RNN reading a sentence word by word, what passes information from one step to the next?

  1. A.The hidden state vector
  2. B.The full list of past inputs
  3. C.The gradient of the loss
  4. D.The vocabulary size
Show answer

Answer: A. The hidden state vector

At each step, an RNN combines the current input with the previous hidden state to produce a new hidden state. That vector is the model's running summary of what it has read so far.

9. How BERT is pretrained

What is BERT's main pretraining objective?

  1. A.Predict the next token from the left only
  2. B.Translate English to French
  3. C.Predict randomly masked tokens from both sides
  4. D.Classify images into 1,000 classes
Show answer

Answer: C. Predict randomly masked tokens from both sides

BERT hides about 15% of tokens and learns to predict them using context on both sides, which makes it a bidirectional encoder good for classification and extraction. GPT-style models instead predict the next token left to right.

10. Slope of the sigmoid at zero

The sigmoid derivative is σ(x)(1 − σ(x)). What is its value at x = 0?

  1. A.0.50
  2. B.0.25
  3. C.1.00
  4. D.0.00
Show answer

Answer: B. 0.25

σ(0) = 0.5, so the derivative is 0.5 × 0.5 = 0.25. This is the largest value the sigmoid slope ever takes, which is one reason gradients shrink through stacked sigmoid layers.

11. Every weight set to zero

You initialise all weights of a hidden layer to the same value (zero). Why does the network learn poorly?

  1. A.All hidden units get identical updates forever
  2. B.Zero weights make the loss value undefined
  3. C.PyTorch refuses to train weights set to zero
  4. D.The learning rate is ignored for zero weights
Show answer

Answer: A. All hidden units get identical updates forever

If every unit starts the same, each gets the same gradient and remains a copy of the others, so the layer acts like a single unit. Random initialisation breaks this symmetry. Biases can start at zero safely.

12. Spotting vanishing gradients

What is a direct way to check whether early layers are getting useful gradients?

  1. A.Print the final validation accuracy
  2. B.Log each layer's gradient norm during training
  3. C.Count the number of parameters
  4. D.Check the size of the saved checkpoint
Show answer

Answer: B. Log each layer's gradient norm during training

After backward(), p.grad.norm() for each layer shows how much signal it receives. Norms that shrink by orders of magnitude towards the input suggest vanishing gradients; huge or growing norms suggest explosion. Tools like TensorBoard can plot these.

13. 800 labelled images

You have 800 labelled photos of shop shelves for a 5-class problem. What is the best starting approach?

  1. A.Train a large CNN from random weights
  2. B.Fine-tune a pretrained model with augmentation
  3. C.Use a 50-layer MLP on raw pixels
  4. D.Wait until you have a million images
Show answer

Answer: B. Fine-tune a pretrained model with augmentation

Pretrained models already know general visual features, so a small labelled set is enough to adapt them. Augmentation adds variety. Training from scratch on 800 images would overfit badly.

14. NaN from a hand-written log loss

A custom loss computes -(y * torch.log(p)).sum() and turns NaN when some p values reach exactly 0. What is the simplest fix?

  1. A.Divide the loss by the batch size
  2. B.Use a smaller batch size
  3. C.Use log-softmax on logits, or clamp p
  4. D.Remove the minus sign
Show answer

Answer: C. Use log-softmax on logits, or clamp p

log(0) is −inf, and 0 × −inf gives NaN. Working with logits and log_softmax (or F.cross_entropy) avoids computing log of a rounded-off zero. Clamping p to a small minimum is a quick patch.

15. Dot product of two vectors

What is the dot product of the vectors a = [1, 2, 3] and b = [4, 5, 6]?

  1. A.21
  2. B.32
  3. C.15
  4. D.36
Show answer

Answer: B. 32

The dot product multiplies matching elements and adds the results: 1×4 + 2×5 + 3×6 = 4 + 10 + 18 = 32. It is the core operation behind similarity search and attention scores.

16. Length of a vector

What is the L2 norm (Euclidean length) of the vector [3, 4]?

  1. A.7
  2. B.12
  3. C.25
  4. D.5
Show answer

Answer: D. 5

The L2 norm is the square root of the sum of squared elements: sqrt(9 + 16) = sqrt(25) = 5. Dividing a vector by its norm gives a unit vector, which is how embeddings are normalised.

17. A learning rate that is too high

You train a model with a very high learning rate. What are you most likely to see?

  1. A.The loss decreases smoothly but very slowly
  2. B.The model trains perfectly in one step
  3. C.The loss jumps around or blows up instead of settling
  4. D.The model uses less GPU memory
Show answer

Answer: C. The loss jumps around or blows up instead of settling

Gradient descent steps of size learning rate × gradient. If the steps are too big, updates overshoot the minimum, so the loss oscillates or diverges, sometimes to NaN. A too-small learning rate gives the slow, smooth case instead.

18. Reading a perplexity number

Model A has a perplexity of 8 on a held-out test set and model B has 15 on the same set with the same tokenizer. What does this tell you?

  1. A.B predicts the test text better because higher perplexity means more knowledge
  2. B.A writes text that is about twice as long as B's
  3. C.A predicts the test text better; it is less surprised by each next token
  4. D.Nothing, because perplexity only measures training speed
Show answer

Answer: C. A predicts the test text better; it is less surprised by each next token

Perplexity is the exponential of the average cross-entropy loss. Roughly, a perplexity of 8 means the model is as uncertain as choosing among 8 equally likely tokens. Lower is better, but it is only comparable across models that use the same tokenizer and test data.

19. Why training uses bf16

Most LLM training today uses bfloat16 (bf16) rather than float16 (fp16). What is the main advantage of bf16?

  1. A.It stores more decimal digits of precision than fp32
  2. B.It uses one byte per value instead of two
  3. C.It only works on CPUs, which are cheaper than GPUs
  4. D.It has the same exponent range as fp32, so values rarely overflow
Show answer

Answer: D. It has the same exponent range as fp32, so values rarely overflow

bf16 keeps fp32's 8 exponent bits and gives up mantissa bits, so it can represent very large and very small numbers with less precision. fp16 has only 5 exponent bits and overflows above about 65,504. Both use 2 bytes per value.

20. DataParallel or DistributedDataParallel

PyTorch has nn.DataParallel and DistributedDataParallel (DDP). Why is DDP recommended even on one machine?

  1. A.DataParallel cannot run on more than one GPU
  2. B.DDP uses one process per GPU, avoiding a bottleneck
  3. C.DDP needs no communication between GPUs
  4. D.DataParallel is only for CPUs
Show answer

Answer: B. DDP uses one process per GPU, avoiding a bottleneck

DataParallel uses one process and thread per GPU, scattering inputs and gathering outputs on GPU 0 each step, which is limited by Python's GIL and GPU 0. DDP gives each GPU its own process and overlaps gradient all-reduce with backward, so it scales better.

Deep Learning intermediate interview questions medium

21. Different results every inference run

A PyTorch model gives slightly different predictions each time you run it on the same input, and validation accuracy is lower than expected. The model has dropout and batch norm layers. What did you most likely forget?

model.load_state_dict(torch.load("model.pt"))
with torch.no_grad():
    preds = model(x_val)
  1. A.Calling model.eval() before inference
  2. B.Calling optimizer.zero_grad()
  3. C.Calling loss.backward()
  4. D.Setting requires_grad=True on the inputs
Show answer

Answer: A. Calling model.eval() before inference

Without model.eval(), dropout stays active (random outputs) and batch norm uses the current batch statistics instead of the running averages. torch.no_grad() only stops gradient tracking; it does not switch layers to inference mode.

22. Conv output size with stride 2

A 224 x 224 image passes through a convolution with a 7 x 7 kernel, stride 2 and padding 3. What is the output height and width?

  1. A.109
  2. B.112
  3. C.115
  4. D.224
Show answer

Answer: B. 112

Output size = floor((W + 2P - K) / S) + 1 = floor((224 + 6 - 7) / 2) + 1 = floor(111.5) + 1 = 112. This is the classic first layer of ResNet-style networks, which halves the resolution.

23. Softmax before CrossEntropyLoss

What is wrong with this classification loss code?

probs = torch.softmax(logits, dim=1)
loss = nn.CrossEntropyLoss()(probs, targets)  # targets: class indices
  1. A.CrossEntropyLoss applies log-softmax itself, so pass raw logits
  2. B.Nothing, applying softmax first is the recommended pattern
  3. C.The targets must be one-hot float vectors for this loss
  4. D.The softmax must use dim=0 when working with batches
Show answer

Answer: A. CrossEntropyLoss applies log-softmax itself, so pass raw logits

nn.CrossEntropyLoss combines log-softmax and negative log-likelihood. Feeding it probabilities applies softmax twice, which squashes gradients and slows or stalls learning. It accepts class indices as targets, so one-hot vectors are not required.

24. Where scheduler.step() goes

You use StepLR(optimizer, step_size=30, gamma=0.1) and intend to cut the LR every 30 epochs. Where should scheduler.step() be called?

  1. A.Once per batch, before optimizer.step()
  2. B.Once per epoch, after the optimizer steps
  3. C.Once, before training starts
  4. D.Inside the loss function
Show answer

Answer: B. Once per epoch, after the optimizer steps

StepLR counts calls to step(), so for an epoch-based schedule call it once per epoch, after that epoch's optimizer.step() calls. Calling it per batch would decay the LR after 30 batches. Per-iteration schedulers like OneCycleLR are stepped every batch instead.

25. Why GPT uses a causal mask

Why does a decoder-only language model mask attention so position i can only attend to positions ≤ i?

  1. A.To halve the number of model parameters
  2. B.To let the model read text right to left
  3. C.So it cannot see the tokens it must predict
  4. D.To shrink the vocabulary size it needs
Show answer

Answer: C. So it cannot see the tokens it must predict

The model is trained to predict each next token. If position i could see position i + 1, it would copy the answer and learn nothing useful. The mask sets future scores to −∞ before softmax, giving them zero weight.

26. Active parameters in a mixture of experts

A mixture-of-experts layer has 8 experts and routes each token to the top 2. Compared with one dense expert of the same size, about how much feed-forward compute does each token use?

  1. A.About 8 experts' worth
  2. B.About 1 expert's worth
  3. C.About 16 experts' worth
  4. D.About 2 experts' worth
Show answer

Answer: D. About 2 experts' worth

Only the 2 selected experts run for each token, so compute per token is about twice one expert, while total parameters are 8 experts' worth. MoE grows capacity without growing per-token compute in proportion, at the cost of memory and routing complexity.

27. Upsampling with a transposed conv

nn.ConvTranspose2d(64, 32, kernel_size=2, stride=2) is applied to a (1, 64, 16, 16) tensor. What is the output shape?

  1. A.(1, 32, 32, 32)
  2. B.(1, 32, 8, 8)
  3. C.(1, 64, 32, 32)
  4. D.(1, 32, 16, 16)
Show answer

Answer: A. (1, 32, 32, 32)

For a transposed conv, output size = (in − 1) × stride − 2 × padding + kernel = 15 × 2 + 2 = 32. Channels go from 64 to 32. It learns to upsample, and is common in decoders, though resize-then-conv avoids checkerboard artefacts.

28. Many boxes on one car

A detector outputs six overlapping boxes around the same car with scores 0.9 down to 0.6. What step removes the duplicates?

  1. A.Batch normalisation
  2. B.Non-maximum suppression
  3. C.Data augmentation
  4. D.Label smoothing
Show answer

Answer: B. Non-maximum suppression

NMS keeps the highest-scoring box and removes others that overlap it beyond an IoU threshold, then repeats. This leaves one box per object. Some newer detectors (such as DETR) are trained to avoid duplicates and skip NMS.

29. Setting cudnn.benchmark = True

When does torch.backends.cudnn.benchmark = True usually speed up training?

  1. A.When input shapes stay the same every batch
  2. B.When every batch has a different size
  3. C.When training on the CPU
  4. D.When the model has no convolutions
Show answer

Answer: A. When input shapes stay the same every batch

With benchmark on, cuDNN times several convolution algorithms for each new input shape and caches the fastest. Fixed shapes pay that cost once. If shapes change constantly, it keeps re-benchmarking and can slow things down. It can also reduce determinism.

30. Loss jumps after resuming training

You resume a run from a checkpoint containing only model.state_dict(). Loss spikes for a while before recovering. What was missing?

  1. A.The DataLoader's batch size
  2. B.Optimiser and scheduler state
  3. C.The random seed used for dropout
  4. D.The number of layers in the model
Show answer

Answer: B. Optimiser and scheduler state

A fresh Adam starts with zero moment estimates and a reset bias correction, and the scheduler may restart warmup or the peak LR. Saving and loading optimizer.state_dict(), scheduler.state_dict(), the step count and the GradScaler gives a smooth resume.

31. Keeping an exponential moving average of weights

Some training recipes keep an exponential moving average (EMA) of the model weights and evaluate with it. Why?

  1. A.It cuts the training memory roughly in half
  2. B.It removes the need for an optimiser at all
  3. C.Averaged weights are smoother and often score better
  4. D.It lets the model train on more data per step
Show answer

Answer: C. Averaged weights are smoother and often score better

Weights from the final steps bounce around because of gradient noise. Averaging them, with decay such as 0.999, gives a smoother model that often generalises better. It is common in diffusion models and semi-supervised learning.

32. Reading a layer's output

You want to capture the output of model.layer3 during a forward pass without changing the model code. What is the standard tool?

  1. A.register_forward_hook on model.layer3
  2. B.torch.no_grad() around the call
  3. C.model.eval() before the pass
  4. D.Calling layer3 with retain_graph=True
Show answer

Answer: A. register_forward_hook on model.layer3

A forward hook is a function called with (module, input, output) every time the module runs, so you can store or inspect the output. Remove it with the handle's remove() when done. It is handy for feature extraction and debugging.

33. Gradient size of L1 and L2 losses

For a single prediction with error e = prediction − target, how do the gradients of L1 loss |e| and L2 loss e² compare as e shrinks towards 0?

  1. A.Both shrink in proportion to e
  2. B.L1 stays ±1; L2 shrinks with e
  3. C.L1 shrinks with e; L2 stays ±1
  4. D.Both stay constant as e shrinks
Show answer

Answer: B. L1 stays ±1; L2 shrinks with e

d|e|/de = sign(e), a constant ±1, while d(e²)/de = 2e, which goes to zero with the error. So L2 takes gentle steps near the target and big ones for outliers; L1 is robust to outliers but can oscillate near zero. Smooth L1 (Huber) blends them.

34. Same data order every epoch

With DistributedSampler(shuffle=True), each GPU sees its batches in exactly the same order every epoch. What was forgotten?

  1. A.Setting num_workers above zero
  2. B.Calling model.train() each epoch
  3. C.Calling sampler.set_epoch(epoch)
  4. D.Passing drop_last=False to the loader
Show answer

Answer: C. Calling sampler.set_epoch(epoch)

DistributedSampler seeds its shuffle with the seed plus the epoch number. Without set_epoch, the epoch stays 0, so the shuffle is identical every time. Calling it at the start of each epoch gives a new order that all ranks agree on.

35. Compute-optimal training tokens

The Chinchilla scaling result suggests about 20 training tokens per parameter for compute-optimal training. How many tokens is that for a 7B model?

  1. A.About 140 billion
  2. B.About 7 billion
  3. C.About 1.4 trillion
  4. D.About 35 billion
Show answer

Answer: A. About 140 billion

20 × 7 billion = 140 billion tokens. Chinchilla found that many earlier models were undertrained for their size. Note that compute-optimal is not the same as best for deployment; many modern small models are trained on far more tokens.

36. Why training starts with warmup

LLM training schedules usually raise the learning rate linearly from near zero over the first few thousand steps. Why?

  1. A.Early updates are noisy and Adam's estimates are poor, so big steps can diverge
  2. B.The GPUs need time to heat up before they reach full speed
  3. C.The tokenizer is still being trained during the first steps
  4. D.A low starting rate makes the model memorise the first batches
Show answer

Answer: A. Early updates are noisy and Adam's estimates are poor, so big steps can diverge

At the start, weights are random, gradients are large and Adam's moment estimates are built from very few steps. A full learning rate then can blow up the loss. Warmup lets the model and optimiser settle before cosine or linear decay takes over.

37. Writing a custom fused kernel

Your team needs a custom fused kernel but has little CUDA C++ experience. Which tool lets you write GPU kernels in Python-like code with block-level operations?

  1. A.NumPy
  2. B.OpenAI Triton
  3. C.Pandas
  4. D.Numba on CPU only
Show answer

Answer: B. OpenAI Triton

Triton is a Python-based language and compiler for GPU kernels. You write operations on blocks of data and it handles many low-level details such as memory coalescing. torch.compile also generates Triton kernels under the hood.

38. TensorRT engine fails on new GPUs

A TensorRT engine built on an A10G is copied to a node with L4 GPUs and fails or runs poorly. Why?

  1. A.TensorRT only runs on CPUs
  2. B.Engines are built for a specific GPU type and version
  3. C.L4 GPUs cannot run neural networks
  4. D.The model weights were deleted in the copy
Show answer

Answer: B. Engines are built for a specific GPU type and version

TensorRT picks kernels and layouts for the exact GPU architecture, plus TensorRT and CUDA versions, when it builds an engine. Build engines in CI for each target GPU type, or rebuild at deploy time.

39. First requests after deploy are slow

After deploying a model wrapped in torch.compile, the first few requests take 40 seconds, then latency is normal. What should the deployment do?

  1. A.Run warm-up requests before marking the pod ready
  2. B.Remove all timeouts for the first requests
  3. C.Restart the pod after each request
  4. D.Raise the request batch size
Show answer

Answer: A. Run warm-up requests before marking the pod ready

torch.compile compiles kernels on first use, and new input shapes can trigger recompiles. Sending representative warm-up inputs before the readiness probe passes moves that cost out of user traffic. Caching compiled artefacts can also help.

40. Certifying the customer admin

Why do many SaaS vendors run an admin certification for each customer's system administrator?

  1. A.Certification is legally needed to use the product
  2. B.Certified admins are charged a higher seat price
  3. C.It lets the vendor stop all other training
  4. D.A skilled admin solves issues in-house and drives adoption
Show answer

Answer: D. A skilled admin solves issues in-house and drives adoption

An admin who knows configuration, permissions and best practice handles most day-to-day questions inside the customer. That lowers support load and keeps the account healthy even when the CSM is not involved.

Deep Learning interview questions for experienced candidates hard

41. Lighting wrong on stretched mesh

A sphere scaled (2, 1, 1) has wrong lighting in a custom shader that transforms normals with the model matrix. What is the correct normal transform?

  1. A.The model matrix itself, then skip normalising
  2. B.The view-projection matrix
  3. C.The inverse transpose of the model matrix's upper 3x3, then normalise
  4. D.The transpose of the projection matrix
Show answer

Answer: C. The inverse transpose of the model matrix's upper 3x3, then normalise

Under non-uniform scale, transforming a normal like a position tilts it so it is no longer perpendicular to the surface. The inverse transpose keeps normals perpendicular. For rotation and uniform scale only, the model matrix works, which is why the bug appears only on stretched objects.

42. Why fp16 needs loss scaling but bf16 usually does not

Mixed precision training with fp16 often needs dynamic loss scaling, while bf16 training usually works without it. Why?

  1. A.bf16 has more mantissa bits than fp32
  2. B.fp16 does not support matrix multiplication on GPUs
  3. C.bf16 has the same exponent range as fp32, so small gradients do not underflow to zero
  4. D.bf16 automatically averages gradients across devices
Show answer

Answer: C. bf16 has the same exponent range as fp32, so small gradients do not underflow to zero

fp16 has only 5 exponent bits, so very small gradient values underflow to zero; loss scaling multiplies the loss to shift gradients into range. bf16 keeps fp32 8 exponent bits (same range) but fewer mantissa bits, so underflow is rare, at the cost of lower precision.

43. Why Adam divides by 1 - beta^t

Adam uses m_hat = m / (1 - beta1^t) and v_hat = v / (1 - beta2^t). Why?

  1. A.It applies weight decay to the moment estimates
  2. B.The moments start at zero, so early estimates are biased low
  3. C.It clips any gradient that is too large
  4. D.It anneals the learning rate to zero by the end
Show answer

Answer: B. The moments start at zero, so early estimates are biased low

Both moving averages are initialised at zero, so in the first steps they underestimate the true mean and variance of the gradient. Dividing by (1 - beta^t) corrects that bias, and the correction fades as t grows. Without it, v would be tiny early and the first updates could be badly scaled.

44. Gradient shrink through ten sigmoids

The sigmoid derivative is at most 0.25. Ignoring weights, at most what factor can a gradient keep after passing back through 10 sigmoid layers?

  1. A.About 1 in 40
  2. B.About 1 in a million
  3. C.About a quarter
  4. D.About 1 in a thousand
Show answer

Answer: B. About 1 in a million

0.25¹⁰ = 1 / 4¹⁰ = 1 / 1,048,576, about one millionth. This is the vanishing gradient problem in its simplest form: early layers get almost no learning signal. ReLU-family activations, residual connections and normalisation help.

45. Keys prefixed with module.

load_state_dict fails: missing keys "fc.weight", unexpected keys "module.fc.weight". The checkpoint was saved from a DistributedDataParallel model. What is the fix?

  1. A.Pass strict=False and ignore the warning
  2. B.Retrain the whole model without DDP
  3. C.Strip the "module." prefix from the keys
  4. D.Load the checkpoint onto the CPU first
Show answer

Answer: C. Strip the "module." prefix from the keys

DDP wraps the model, so its state_dict keys gain a "module." prefix. Saving model.module.state_dict() avoids it, or you can rename keys when loading. strict=False would load nothing useful, leaving random weights.

46. Calling backward twice

You call loss.backward() and then loss.backward() again on the same graph. PyTorch raises "Trying to backward through the graph a second time". Why?

  1. A.Gradients can only be computed once per model
  2. B.The optimiser locks the graph
  3. C.The loss is detached after the first call
  4. D.The saved tensors are freed after the first backward
Show answer

Answer: D. The saved tensors are freed after the first backward

By default backward() frees the intermediate buffers it used, to save memory. To backpropagate through the same graph again, pass retain_graph=True the first time. Usually the real fix is to run a fresh forward pass.

47. Parameters in a depthwise separable conv

Ignoring biases, a standard 3×3 conv from 64 to 128 channels has 3·3·64·128 = 73,728 weights. How many does a depthwise separable version (3×3 depthwise on 64 channels, then 1×1 to 128) have?

  1. A.73,728
  2. B.576
  3. C.8,768
  4. D.8,192
Show answer

Answer: C. 8,768

Depthwise: one 3×3 filter per input channel = 9 × 64 = 576. Pointwise: 64 × 128 = 8,192. Total 8,768, about 8.4 times fewer. MobileNet-style models use this to run on phones.

48. An augmentation that changes the label

You train a model to read handwritten digits and add random horizontal and vertical flips. Accuracy on 6 and 9 drops sharply. Why?

  1. A.Flips make the images too large
  2. B.Flips turn some digits into others
  3. C.Flips need a much larger batch size
  4. D.Flipped images cannot be normalised
Show answer

Answer: B. Flips turn some digits into others

Augmentations must keep the label true. Rotating or flipping a 6 can make it look like a 9 while keeping the label 6, which teaches the model contradictions. Choose transformations that are realistic for the task.

49. Padding in a batch of sentences

Sentences of length 5 to 60 are zero-padded to 60 in one batch. Why use pack_padded_sequence before an LSTM?

  1. A.So the LSTM skips pads and keeps true final states
  2. B.So the whole batch fits on a single GPU
  3. C.So padding tokens get an embedding of their own
  4. D.So the LSTM can run without any gradients
Show answer

Answer: A. So the LSTM skips pads and keeps true final states

Without packing, a 5-word sentence's final hidden state comes after 55 pad steps, which corrupts it and wastes compute. Packing tells the RNN each sequence's real length, so it stops at the right point.

50. Sharing embedding and output weights

Many language models tie the output projection to the input embedding matrix. What is the main effect?

  1. A.The model can only output the input tokens
  2. B.Training becomes non-differentiable
  3. C.Fewer parameters, often with equal or better quality
  4. D.The vocabulary must double in size
Show answer

Answer: C. Fewer parameters, often with equal or better quality

Both matrices are vocab × d_model, so sharing them removes one large copy. Using the same vectors to read and write tokens also acts as a useful constraint, and quality usually holds or improves.

51. The reparameterisation trick

In a variational autoencoder, why is z sampled as z = μ + σ·ε with ε ~ N(0, 1) rather than drawn directly from N(μ, σ²)?

  1. A.So z is always positive
  2. B.So gradients can flow through μ and σ
  3. C.So the decoder needs no weights
  4. D.So the KL term becomes zero
Show answer

Answer: B. So gradients can flow through μ and σ

Sampling directly is not differentiable with respect to μ and σ. Moving the randomness into ε makes z a deterministic function of μ and σ plus independent noise, so backpropagation works.

52. Reach of sliding-window attention

Each layer of a model uses sliding-window attention over the previous 4,096 tokens. Stacking 32 such layers, roughly how far back can information travel to the last position?

  1. A.Exactly 4,096 tokens
  2. B.About 8,192 tokens
  3. C.Unlimited, at no cost
  4. D.About 131,000 tokens
Show answer

Answer: D. About 131,000 tokens

Each layer lets information move up to one window back, so through L layers the reach is about L × W = 32 × 4,096 ≈ 131,072 tokens. Per-layer cost stays linear in sequence length, but information from far away is passed indirectly and can weaken.

53. Attention to padding tokens

A batch of sentences is padded to the same length, but no padding mask is passed to the encoder's self-attention. What goes wrong?

  1. A.Tokens attend to pads, so output depends on padding
  2. B.Pad tokens are dropped from the batch automatically
  3. C.The model refuses to run with mixed lengths
  4. D.Only the loss changes, never the outputs
Show answer

Answer: A. Tokens attend to pads, so output depends on padding

Without a key padding mask, pad positions get attention weight like any other token, so the same sentence gives different outputs depending on how much padding its batch has. The mask sets pad scores to −∞ so they get zero weight.

54. Gradient of softmax cross-entropy

Softmax outputs are p = [0.7, 0.2, 0.1] and the true class is index 0. What is the gradient of the cross-entropy loss with respect to the three logits?

  1. A.[−0.3, 0.2, 0.1]
  2. B.[0.3, −0.2, −0.1]
  3. C.[−0.7, 0.2, 0.1]
  4. D.[0.7, 0.2, 0.1]
Show answer

Answer: A. [−0.3, 0.2, 0.1]

For softmax followed by cross-entropy, ∂L/∂z = p − y, where y is the one-hot target [1, 0, 0]. That gives [0.7 − 1, 0.2, 0.1] = [−0.3, 0.2, 0.1]. This simple form is why the two are fused in one loss.

55. Gradient of a linear layer's weights

A layer computes y = Wx with x of size 3 and y of size 2, so W is 2 × 3. Given the upstream gradient δ = ∂L/∂y (size 2), what is ∂L/∂W?

  1. A.δ xᵀ, a 2 × 3 matrix
  2. B.xᵀ δ, a single number
  3. C.Wᵀ δ, a vector of size 3
  4. D.x δᵀ, a 3 × 2 matrix
Show answer

Answer: A. δ xᵀ, a 2 × 3 matrix

Each weight W_ij affects y_i through x_j, so ∂L/∂W_ij = δ_i x_j, which is the outer product δxᵀ with the same 2 × 3 shape as W. Wᵀδ is the gradient passed back to x, not to W.

56. Adam and a scaled-up loss

You multiply the loss by 1,000 and keep Adam's learning rate the same. Ignoring eps, what happens to the size of Adam's parameter updates?

  1. A.1,000 times larger
  2. B.Roughly unchanged
  3. C.1,000 times smaller
  4. D.About 31 times larger
Show answer

Answer: B. Roughly unchanged

Adam divides the gradient's moving average by the square root of the squared-gradient average. Scaling every gradient by c scales both by c, so they cancel. Plain SGD updates would be 1,000 times larger.

57. Test error falls again with more parameters

As model size grows, test error first drops, then rises near the point where the model can just fit the training data, then drops again for much larger models. What is this called?

  1. A.Double descent
  2. B.Catastrophic forgetting
  3. C.Mode collapse
  4. D.Gradient explosion
Show answer

Answer: A. Double descent

Double descent shows the classic U-shaped bias-variance curve is only part of the story. Around the interpolation threshold, models fit noise badly; much larger models can fit the data in smoother ways and generalise better. It can also appear over training epochs.

58. Writing a custom autograd Function

In a torch.autograd.Function subclass, what must backward(ctx, grad_output) return?

  1. A.The loss value for the batch
  2. B.One gradient per forward input, None for non-tensors
  3. C.The updated parameters
  4. D.A single scalar summarising all gradients
Show answer

Answer: B. One gradient per forward input, None for non-tensors

backward receives the gradient of the loss with respect to the output and must return the gradient with respect to each input of forward, in order. Inputs that need no gradient, such as ints or flags, get None. Use gradcheck to verify it.

59. Multi-GPU job that hangs forever

A distributed job hangs with no error. Code does `if loss < threshold: dist.all_reduce(metric)`, and loss differs between ranks. What is wrong?

  1. A.all_reduce cannot sum floating-point values
  2. B.Some ranks enter the collective and others never do
  3. C.The threshold must be the same type as loss
  4. D.NCCL only supports 2 GPUs
Show answer

Answer: B. Some ranks enter the collective and others never do

Collective operations need every rank to call them in the same order. If only some ranks take the branch, they wait forever for the others. Make the decision identical on all ranks (for example, all-reduce the condition first), and set NCCL timeouts and debug logging to catch such hangs.

60. Training past compute-optimal

Llama 3 8B was trained on about 15 trillion tokens, far beyond the ~160B that Chinchilla's rule suggests. Why would a lab do this?

  1. A.A smaller model trained longer is cheaper to serve at a given quality level
  2. B.Chinchilla's rule only applies to encoder-only models
  3. C.Training longer always lowers the total training compute
  4. D.Extra tokens are needed to fill the tokenizer's vocabulary
Show answer

Answer: A. A smaller model trained longer is cheaper to serve at a given quality level

Chinchilla minimises training compute for a target loss, but it ignores inference. If a model will serve billions of requests, a smaller one trained on many more tokens can match a bigger model's quality while costing far less per token to run. Loss keeps improving slowly well past 20 tokens per parameter.

Ready to test yourself?

Take a 15-question Deep Learning MCQ test with a score at the end, or practise the full Deep Learning question bank and earn points on the leaderboard.

Start the Deep Learning testTake a mock interview