Machine Learning Engineer
Model evaluation questions
85 questions on Model evaluation for machine learning engineers, easiest first.
Easy 22
- Big gap between train and validation accuracyeasy
- Why keep a separate test seteasy
- Definition of precisioneasy
- Accuracy on imbalanced dataeasy
- Recall from confusion countseasy
- RMSE from four errorseasy
- How often a row is validated in k-foldeasy
- Why stratify the foldseasy
- Specificity by handeasy
- The x-axis of an ROC curveeasy
- MAE of three errorseasy
- MAPE by handeasy
- R-squared by handeasy
- Lowering the decision thresholdeasy
- Top-5 accuracyeasy
- First number to compare againsteasy
- Recall at 10 by handeasy
- Score versus tree deptheasy
- Accuracy from confusion countseasy
- Reporting cross-validation resultseasy
- Test set from a later periodeasy
- Checking a 90% prediction intervaleasy
Medium 39
- Scaling before splittingmedium
- Validating a forecasting modelmedium
- Handling class imbalancemedium
- Probabilities that cannot be trustedmedium
- F1 score from a confusion matrixmedium
- Log loss of a poor predictionmedium
- AUC from four scoresmedium
- Precision at 5 for a ranked listmedium
- NDCG for a three-item rankingmedium
- Target encoding without leakagemedium
- Several scans per patientmedium
- Leave-one-out on a big datasetmedium
- Test split for ten million rowsmedium
- MAPE on low-selling productsmedium
- Why use adjusted R-squaredmedium
- ROC-AUC looks great, alerts are uselessmedium
- Weighted F1 on imbalanced classesmedium
- Choosing F2 instead of F1medium
- Which class gets confusedmedium
- Points below the diagonalmedium
- Accuracy after temperature scalingmedium
- Test accuracy that is too highmedium
- Error bars on a test metricmedium
- Hundreds of runs on one validation setmedium
- MRR across three queriesmedium
- Better offline, flat onlinemedium
- Validation error still fallingmedium
- Good overall, bad for one citymedium
- RMSLE versus RMSEmedium
- Negative scores from cross_val_scoremedium
- What best_score_ meansmedium
- Precision and recall at 0.6medium
- Balanced accuracymedium
- Gini coefficient in credit scoringmedium
- Reading a MASE of 0.8medium
- Running a model in shadow modemedium
- Picking a threshold with a recall floormedium
- Scoring clusters against known labelsmedium
- Scoring invoice field extractionmedium
Hard 24
- ROC-AUC after calibrationhard
- A feature that is too goodhard
- SMOTE applied before cross-validationhard
- Negative R-squared on test datahard
- Validation loss and accuracy both risehard
- A rolling mean that peeks aheadhard
- When nested cross-validation is neededhard
- Macro F1 with a weak minority classhard
- A metric using all four cellshard
- Agreement beyond chancehard
- Expected calibration error by handhard
- Calibrating with only 300 rowshard
- Choosing features before cross-validatinghard
- A gap between train and test windowshard
- Comparing two models on one test sethard
- Is a 1-point gain real?hard
- Threshold from error costshard
- Quantile loss of one forecasthard
- Only approved loans have outcomeshard
- What the KS statistic measureshard
- Fraud labels that arrive latehard
- Twenty metrics, one significanthard
- Which metric is stable when base rate shiftshard
- Average prediction versus observed ratehard