Skip to content
RED SAGSkills
mediumData Scientist interview question

You are building a model where only 0.5% of examples are positive. How do you approach training and evaluation?

What the interviewer wants to hear

  • Use metrics like PR-AUC, recall at a fixed precision, not accuracy
  • Keep the real class balance in validation and test sets
  • Try class weights or resampling only inside training folds
  • Tune the decision threshold to business costs
  • Check calibration if probabilities are used downstream

Model answer

First I would make sure evaluation is honest. Accuracy is useless at 0.5% positives, so I would use PR-AUC and a business metric such as recall at the precision the operations team can handle. I would use stratified splits, or time-based ones if relevant, and keep the true class balance in validation and test. For training, I would start with a strong baseline like gradient boosting with class weights, which often works as well as resampling. If I try undersampling or SMOTE, I apply it only inside training folds to avoid leakage. Then I choose the threshold from the precision-recall curve based on the cost of a missed positive versus a false alarm. If the scores feed pricing or prioritisation, I check calibration, because weighting and resampling distort probabilities, and recalibrate with isotonic or Platt scaling if needed.

Can you answer it under time pressure?

Take a timed medium round and get scored on every answer.

Start mock interview

More medium Data Scientist questions