Your neural network loss is not decreasing. How do you debug it?
What the interviewer wants to hear
- Check data and labels, input pipeline, normalisation
- Try to overfit a tiny batch
- Check learning rate, initialisation, loss function choice
- Inspect gradients for vanishing or exploding values
- Remove complexity and compare against a known baseline
Model answer
I start with the data. I look at a few batches to confirm inputs and labels match, are normalised and are not all the same, and that augmentation has not broken them. Next I try to overfit a tiny batch of 10 to 50 examples. If the model cannot drive loss close to zero on that, there is a bug in the model, loss or training loop. I check common mistakes: wrong loss for the task, applying softmax twice, forgetting optimizer.zero_grad(), or a frozen layer. Then I sweep the learning rate, since too high or too low are both common causes. I log gradient norms per layer to spot vanishing or exploding gradients, and use gradient clipping or better initialisation if needed. Finally I simplify: smaller model, no regularisation, and compare against a simple baseline to rebuild step by step.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Machine Learning Engineer questions
- You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
- What problems does a feature store solve, and when is it worth adding one?
- How do you decide between batch predictions and real-time (online) inference for a model?
- After deploying a model, what would you monitor, and how do you handle data drift?
- Walk me through how you would build a RAG system to answer questions over a company knowledge base.
- How would you evaluate an LLM-powered feature before and after launch?
- How do you approach hyperparameter tuning for a model on a limited compute budget?
- A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
- Your new model scores far higher than anything before it. How do you check for data leakage?