How do you approach hyperparameter tuning for a model on a limited compute budget?

What the interviewer wants to hear

• Start from known good defaults and a strong baseline
• Tune the most sensitive parameters first, such as the learning rate
• Use random search or Bayesian optimisation over log-scaled ranges
• Use early stopping or successive halving to kill bad trials
• Tune on validation folds and keep the test set untouched

Model answer

I start from sensible defaults from the library or a similar published setup and get a baseline score. Then I focus on the parameters that usually matter most. For neural networks that is the learning rate, then batch size, weight decay and schedule. For gradient boosting it is learning rate, number of trees with early stopping, depth and regularisation. I search on a log scale for things like learning rate and use random search or Bayesian optimisation rather than a grid. To save compute I use early stopping inside each trial and methods like successive halving or Hyperband that stop weak trials early. I often tune on a subset of data first, then confirm the best settings on the full data. All decisions use validation or cross-validation, and the test set is used once at the end.

More medium Machine Learning Engineer questions

• You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
• What problems does a feature store solve, and when is it worth adding one?
• How do you decide between batch predictions and real-time (online) inference for a model?
• Your neural network loss is not decreasing. How do you debug it?
• After deploying a model, what would you monitor, and how do you handle data drift?
• Walk me through how you would build a RAG system to answer questions over a company knowledge base.
• How would you evaluate an LLM-powered feature before and after launch?
• A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
• Your new model scores far higher than anything before it. How do you check for data leakage?