PyTorch interview questions and answers

1. Logging loss without leaking memory

A training loop does total += loss each step to track the epoch loss, and GPU memory keeps rising. What is the usual fix?

1. A.Call torch.cuda.empty_cache() each step
2. B.Use a larger batch size
3. C.Call loss.backward() twice
4. D.Use total += loss.item()

2. Tensors on different devices

You call model.to("cuda") but forget to move the batch. What happens on model(x) with x on the CPU?

1. A.PyTorch silently copies x over to the GPU
2. B.The model silently falls back to the CPU
3. C.Training runs, but all gradients are zero
4. D.A RuntimeError about tensors on two devices

3. model(x) versus model.forward(x)

Why should you call model(x) rather than model.forward(x)?

1. A.forward() skips the gradient calculation
2. B.forward() always runs on the CPU
3. C.They differ only in speed
4. D.model(x) also runs registered hooks

4. Default betas in Adam

What are PyTorch's default (beta1, beta2) values for torch.optim.Adam?

1. A.(0.9, 0.999)
2. B.(0.5, 0.5)
3. C.(0.99, 0.9)
4. D.(0.1, 0.01)

5. Adding weight decay to an optimiser

What does passing weight_decay=0.01 to torch.optim.AdamW do?

1. A.Lowers the learning rate every epoch
2. B.Drops 1% of activations at random
3. C.Shrinks every weight a little at each step
4. D.Clips gradients to a norm of 0.01

6. Size after nn.Flatten

A conv stack outputs (N, 64, 7, 7). After nn.Flatten(), what in_features must the next nn.Linear have?

1. A.3,136
2. B.448
3. C.3,584
4. D.4,096

7. What ToTensor does to pixels

torchvision.transforms.ToTensor() is applied to an 8-bit RGB PIL image. What does it return?

1. A.An int (H, W, C) tensor with values 0–255
2. B.A float (C, H, W) tensor with values in [0, 1]
3. C.A float (H, W, C) tensor in [−1, 1]
4. D.A float (C, H, W) tensor in [0, 255]

8. Size of an embedding table

How many parameters does nn.Embedding(50000, 768) have?

1. A.38.4 million
2. B.50,768
3. C.768
4. D.3.84 million

9. Up-weighting a rare class

In a 3-class problem, class 2 is rare. How can you make its errors count more in nn.CrossEntropyLoss?

1. A.Set reduction="sum" on the loss
2. B.Use label_smoothing=0.5 in the loss
3. C.Pass weight=torch.tensor([1., 1., 5.])
4. D.Call loss.backward() five times per step

10. What DDP syncs between GPUs

In DistributedDataParallel training on 4 GPUs, what is synchronised between the processes during each backward pass?

1. A.Gradients, averaged with all-reduce
2. B.The input batches
3. C.The loss values only
4. D.Nothing until the end of the epoch

11. DataParallel or DistributedDataParallel

Why does PyTorch recommend DistributedDataParallel over nn.DataParallel even on a single machine with several GPUs?

1. A.DataParallel cannot use more than two GPUs
2. B.DDP uses one process per GPU and avoids a bottleneck
3. C.DDP needs no changes to the training script
4. D.DataParallel gives wrong gradients

12. Tensors on different devices

This code raises "Expected all tensors to be on the same device". What is the fix?

model = MyNet().to("cuda")
x = torch.randn(16, 128)
out = model(x)

1. A.Move the model back with model.cpu() only after
2. B.Move the input with x = x.to("cuda")
3. C.Wrap the call in torch.no_grad()
4. D.Set the batch size to 1

13. Memory during evaluation

Your evaluation loop uses far more GPU memory than expected. Wrapping it in torch.no_grad() (or torch.inference_mode()) fixes it. Why?

1. A.The model weights are moved to the CPU
2. B.Activations are not kept for a backward pass
3. C.The batch size is halved automatically
4. D.The model is converted to INT8

14. What autocast does

What happens to operations inside a torch.autocast(device_type="cuda", dtype=torch.bfloat16) block?

with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    logits = model(x)
    loss = loss_fn(logits, y)

1. A.Every tensor is permanently converted to BF16
2. B.Matmuls run in BF16; sensitive ops stay in FP32
3. C.The model weights are saved in BF16 on disk
4. D.Only the loss is computed in BF16

15. Purpose of Triton Inference Server

NVIDIA Triton Inference Server is chosen by a team with PyTorch, ONNX and TensorRT models. What does it provide?

1. A.A Python library for writing training loops
2. B.One server that runs models from many frameworks
3. C.A database for storing feature values
4. D.A GPU driver that replaces CUDA

16. Gradient of a sum of squares

What does this PyTorch code print?

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = (x ** 2).sum()
y.backward()
print(x.grad)

1. A.tensor([1., 4., 9.])
2. B.tensor([2., 4., 6.])
3. C.tensor([1., 2., 3.])
4. D.None

17. backward() on a vector

What happens when this code runs?

x = torch.randn(3, requires_grad=True)
y = x * 2
y.backward()

1. A.x.grad becomes tensor([2., 2., 2.]) automatically
2. B.RuntimeError: grad can only be implicitly created for scalars
3. C.x.grad becomes the single value tensor(6.)
4. D.Nothing happens, because y is not called a loss

18. What detach() does

In a training loop, z = model_a(x).detach() is fed into model_b. What happens when you call loss.backward() on model_b's loss?

1. A.Only model_b gets gradients
2. B.Both models get gradients
3. C.Only model_a gets gradients
4. D.backward() raises an error

19. torch.cat or torch.stack

a and b both have shape (4, 8). What are the shapes of torch.cat([a, b], dim=0) and torch.stack([a, b], dim=0)?

1. A.(2, 4, 8) and (8, 8)
2. B.(4, 16) and (2, 4, 8)
3. C.(8, 8) and (2, 4, 8)
4. D.(8, 8) and (4, 8, 2)

20. Double versus float error

Inputs come from a NumPy float64 array: x = torch.from_numpy(arr). model(x) fails with "expected scalar type Float but found Double". What is the fix?

1. A.Convert with x = x.float()
2. B.Call model.double() on every batch
3. C.Convert with x = x.long()
4. D.Wrap the call in torch.no_grad()

21. Loss for a single-logit classifier

Your binary classifier ends with nn.Linear(64, 1) and no activation. Which loss is the right choice?

1. A.nn.BCELoss
2. B.nn.CrossEntropyLoss with 1 class
3. C.nn.BCEWithLogitsLoss
4. D.nn.MSELoss on raw logits

22. Target format for CrossEntropyLoss

Logits have shape (32, 5). For nn.CrossEntropyLoss with hard labels, what should the target tensor be?

1. A.Shape (32,), dtype long, values 0–4
2. B.Shape (32, 5) one-hot, dtype long
3. C.Shape (32, 1), dtype float
4. D.Shape (5,), dtype long

23. Freezing a pretrained backbone

You want to train only the new classifier head on a pretrained ResNet. What is the standard approach?

1. A.Call model.eval() on the backbone only
2. B.Set requires_grad=False on backbone parameters
3. C.Wrap the head in torch.no_grad()
4. D.Use a learning rate of zero for the head

24. Slow first step after torch.compile

After model = torch.compile(model), the first training step takes 90 seconds and later steps are faster than before. What explains the slow first step?

1. A.torch.compile reinitialises all of the weights
2. B.Graph capture and kernel compilation run on first call
3. C.The first batch is always ten times larger
4. D.Compiled models must re-download their weights

25. Loss jumps at each epoch start

Training loss jumps sharply at the start of each epoch, then falls again. The DataLoader has shuffle=False and the data is sorted by class. What is happening?

1. A.The learning rate is reset to its peak at every epoch
2. B.Batches hold one class at a time, so the model keeps shifting
3. C.Dropout is switched off at the start of each epoch
4. D.The model reloads its initial weights at each epoch

26. Using argmax inside a loss

A custom loss uses torch.argmax(logits, dim=1) to pick classes and compares them with labels. The model does not learn. Why?

1. A.argmax gives no useful gradient
2. B.argmax only works on the CPU
3. C.argmax changes the batch size
4. D.argmax makes all logits negative

27. DDP error about unused parameters

A model with an optional auxiliary head (used only on some steps) fails under DDP with "Expected to have finished reduction in the prior iteration". What is the usual fix?

1. A.Lower the learning rate by 10×
2. B.Use a much smaller batch size
3. C.Pass find_unused_parameters=True
4. D.Call zero_grad() twice per step

28. Memory grows each training step

GPU memory rises every step until OOM. Which line is the likely cause?

total = 0
for x, y in loader:
    loss = loss_fn(model(x), y)
    loss.backward()
    opt.step(); opt.zero_grad()
    total += loss

1. A.opt.zero_grad() is called after opt.step()
2. B.loss.backward() is called inside the loop
3. C.loss_fn is created outside the loop
4. D.total += loss keeps each step's graph alive

29. Memory format for CNNs

For mixed-precision CNN training on tensor-core GPUs, why does converting the model and inputs to torch.channels_last often help?

1. A.It halves the number of channels
2. B.It stores images as grayscale
3. C.It turns convolutions into matmuls on the CPU
4. D.Conv kernels on tensor cores prefer the NHWC layout

30. GradScaler skipped a step

With FP16 and torch.cuda.amp.GradScaler, a few steps are skipped early in training. What is happening?

1. A.The optimizer ran out of memory and paused
2. B.The data loader returned an empty batch
3. C.Skipped steps mean the model has converged
4. D.Gradients overflowed, so the step was skipped

31. view() after a transpose

What happens when this code runs?

import torch
x = torch.randn(4, 6)
y = x.t()
z = y.view(24)

1. A.RuntimeError, because y is not contiguous; use reshape or contiguous()
2. B.z has shape (24,) and shares memory with x
3. C.z silently gets shape (6, 4) instead of (24,)
4. D.x is transposed in place and z is a copy

32. What ZeRO stage 3 shards

With DeepSpeed ZeRO stage 3 or PyTorch FSDP full sharding, what is split across data-parallel GPUs?

1. A.Only the optimizer states
2. B.Optimizer states and gradients only
3. C.Parameters, gradients and optimizer states
4. D.Only the input data batches

33. Layers in a plain Python list

In this model, the optimizer never updates the hidden layers. Why?

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = [nn.Linear(64, 64) for _ in range(4)]
        self.out = nn.Linear(64, 1)

1. A.Linear layers of the same size share their weights
2. B.A plain list hides them from parameters(); use ModuleList
3. C.The out layer blocks gradients to the earlier layers
4. D.super().__init__() must be called at the very end

34. Scaling inside dropout

nn.Dropout(p=0.2) is in training mode. An input activation of 5.0 is kept (not dropped). What value comes out?

1. A.5.00
2. B.6.25
3. C.4.00
4. D.1.25

35. Keys prefixed with module.

load_state_dict fails: missing keys "fc.weight", unexpected keys "module.fc.weight". The checkpoint was saved from a DistributedDataParallel model. What is the fix?

1. A.Pass strict=False and ignore the warning
2. B.Retrain the whole model without DDP
3. C.Strip the "module." prefix from the keys
4. D.Load the checkpoint onto the CPU first

36. Error from an in-place operation

backward() raises "one of the variables needed for gradient computation has been modified by an inplace operation". What usually causes it?

1. A.A tensor saved for backward was changed in place
2. B.The loss was a float instead of a tensor
3. C.The model is on the GPU
4. D.zero_grad() was called before backward

37. Why .grad is None for an intermediate

After backward(), w.grad has values but h.grad is None. Why?

w = torch.randn(3, requires_grad=True)
h = w * 2
loss = (h ** 2).sum()
loss.backward()
print(w.grad, h.grad)

1. A.The multiply moved h off the autograd graph
2. B.Only leaf tensors keep .grad unless retain_grad()
3. C.backward() must be called on h directly instead
4. D.h.grad is only filled in when in eval mode

38. Calling backward twice

You call loss.backward() and then loss.backward() again on the same graph. PyTorch raises "Trying to backward through the graph a second time". Why?

1. A.Gradients can only be computed once per model
2. B.The optimiser locks the graph
3. C.The loss is detached after the first call
4. D.The saved tensors are freed after the first backward

39. Padding in a batch of sentences

Sentences of length 5 to 60 are zero-padded to 60 in one batch. Why use pack_padded_sequence before an LSTM?

1. A.So the LSTM skips pads and keeps true final states
2. B.So the whole batch fits on a single GPU
3. C.So padding tokens get an embedding of their own
4. D.So the LSTM can run without any gradients

40. Backpropagating through a gradient

A WGAN-GP loss needs a penalty on the gradient norm of the critic with respect to its input, and that penalty must itself be trained. Which call is needed?

1. A.loss.backward(retain_graph=False)
2. B.torch.no_grad() around the penalty
3. C.x.detach().requires_grad_(False)
4. D.torch.autograd.grad(..., create_graph=True)

41. Writing a custom autograd Function

In a torch.autograd.Function subclass, what must backward(ctx, grad_output) return?

1. A.The loss value for the batch
2. B.One gradient per forward input, None for non-tensors
3. C.The updated parameters
4. D.A single scalar summarising all gradients

42. MSE loss that never improves

A regression model's loss stays high. Prediction shape is (64, 1) and target shape is (64,). What does F.mse_loss(pred, target) actually compute?

1. A.The correct per-example squared error
2. B.An error, because shapes differ
3. C.The loss for the first example only
4. D.A mean over a broadcast 64 × 64 matrix of pairs

43. Loading a model from a public hub

A teammate wants to load a pickled PyTorch checkpoint downloaded from an unknown public repository in a production pipeline. What is the main risk and safer approach?

1. A.Pickle files are slower to load, so convert to CSV
2. B.There is no risk because model files only contain numbers
3. C.The model might be too accurate
4. D.Unpickling can execute arbitrary code; prefer weight-only formats such as safetensors, verify source and checksums or signatures, and scan artifacts before use

44. OOM late in a long evaluation

An evaluation over 200,000 examples runs out of GPU memory near the end, even inside torch.no_grad(). The loop appends logits to a Python list. What is the fix?

1. A.Move results to CPU or keep only needed numbers
2. B.Use a larger learning rate for evaluation
3. C.Call model.train() before evaluating
4. D.Increase the batch size to finish sooner

45. Loading a model file from the internet

A teammate downloads a fine-tuned model from an unknown uploader as a .pkl / older PyTorch .bin file and loads it on a production server. What is the main risk?

1. A.The model will be slightly less accurate than before
2. B.The file will take too much disk space
3. C.Unpickling can run code; prefer safetensors
4. D.Pickle files only work on Windows