You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.

What the interviewer wants to hear

• Use metrics such as PR-AUC, recall at fixed precision, not accuracy
• Class weights, resampling on training data only, or focal loss
• Tune decision threshold using business costs
• Time-based split and watch for label delay
• Error analysis on false positives and negatives

Model answer

First I would agree on the right metric. Accuracy is useless here, so I would use PR-AUC and recall at a fixed precision, or a cost-based metric using the real cost of missed fraud versus blocked good customers. I would split by time, because fraud patterns change and the model will predict the future. I would also check label delay, since chargebacks can arrive weeks later. For training, gradient boosting works well on tabular fraud data. I would handle imbalance with class weights or by undersampling negatives in the training set only, and then correct the probabilities if needed. After training, I would choose a threshold on validation data that matches the review team capacity or cost trade-off. Finally I would do error analysis on false positives and false negatives to find new features, and monitor drift after launch.

More medium Machine Learning Engineer questions

• What problems does a feature store solve, and when is it worth adding one?
• How do you decide between batch predictions and real-time (online) inference for a model?
• Your neural network loss is not decreasing. How do you debug it?
• After deploying a model, what would you monitor, and how do you handle data drift?
• Walk me through how you would build a RAG system to answer questions over a company knowledge base.
• How would you evaluate an LLM-powered feature before and after launch?
• How do you approach hyperparameter tuning for a model on a limited compute budget?
• A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
• Your new model scores far higher than anything before it. How do you check for data leakage?