How does gradient descent work, and what is the role of the learning rate?

What the interviewer wants to hear

• Minimises a loss function by moving parameters against the gradient
• Learning rate sets the step size
• Too high diverges, too low is slow
• Mini-batch SGD and adaptive optimisers like Adam are common

Model answer

Gradient descent is a way to minimise a loss function. We compute the gradient of the loss with respect to each parameter, which points in the direction of steepest increase, and then move the parameters a small step in the opposite direction. We repeat this until the loss stops improving. The learning rate controls the size of each step. If it is too high, updates overshoot and the loss can oscillate or explode. If it is too low, training is very slow and can get stuck. In practice we use mini-batch stochastic gradient descent, computing gradients on small batches, which is faster and adds helpful noise. Optimisers like Adam adapt the step size per parameter, and schedules such as warmup followed by cosine decay often make training more stable.

More easy Machine Learning Engineer questions

• What is overfitting, how do you detect it, and how do you reduce it?
• Explain precision and recall with a real example. When would you favour one over the other?
• What are the training, validation and test sets used for, and how do you split them?
• Why should you build a simple baseline before a complex model? Give examples of good baselines.
• Explain the bias-variance trade-off and how it guides your choice of model.
• What is an embedding, and how would you use embeddings in a product feature?
• When would you choose a tree-based model like gradient boosting over a linear model, and when not?
• How do you handle missing values in a dataset before training a model?
• Explain in simple terms how self-attention works in a transformer.