hard20 pts · first answer countsrankingab-testing

Better offline metric, worse A/B test

A new ranking model improves offline NDCG by 4% on logged click data, but the online A/B test shows lower engagement. Which explanation is most likely?

A.The A/B test must have a bug, because offline metrics are always rightB.NDCG cannot be computed on click dataC.Logged data is biased by the old model (position and selection bias), so offline gains do not reflect real user behaviourD.The new model is too accurate for users

← ROC-AUC after calibration L2 penalty versus AdamW →