MLOps Engineer
Model monitoring and drift questions
153 questions on Model monitoring and drift for mlops engineers, easiest first.
Easy 43
- What data drift iseasy
- What concept drift iseasy
- What training-serving skew iseasy
- Bundling preprocessing with the modeleasy
- What label shift meanseasy
- Common PSI thresholdseasy
- Drift test for payment methodeasy
- Watching prediction distributioneasy
- Missing values jump overnighteasy
- A new category appearseasy
- Average latency looks fineeasy
- Service health basicseasy
- Running a model in shadoweasy
- Fraud flags tripledeasy
- Joining predictions to labelseasy
- Different tokenizer at servingeasy
- Features from three days agoeasy
- Half the rows went missingeasy
- Tracking model ageeasy
- Inputs to an Evidently reporteasy
- Agreeing on input dataeasy
- Champion and challengereasy
- What a drift runbook giveseasy
- Nobody owns the alerteasy
- Which prompt caused iteasy
- Watching the safety filtereasy
- Counts doubled after a retry fixeasy
- Green job, empty outputeasy
- Mixing versions in metricseasy
- Too many charts on the dashboardeasy
- Paging for minor drifteasy
- Nightly run on a golden seteasy
- Drift that comes backeasy
- New phones, worse OCReasy
- GPU mostly idleeasy
- Pods restart every few hourseasy
- Requests suddenly 10x largereasy
- Weekly chart looks calmeasy
- Simple image drift statseasy
- Why track time to first tokeneasy
- Known requests sent on a timereasy
- Planning monitoring earlyeasy
- After a model incidenteasy
Medium 65
- When to retrainmedium
- Labels arrive weeks latermedium
- Computing PSI for two binsmedium
- Reading a KS drift statisticmedium
- What to log for each LLM callmedium
- SQL offline, Java onlinemedium
- Covariate shift or concept driftmedium
- PSI with an empty binmedium
- Swapping PSI inputsmedium
- KL divergence directionmedium
- Why teams like Jensen-Shannonmedium
- Wasserstein distance for a shiftmedium
- Choosing the reference windowmedium
- Hourly drift checks are noisymedium
- Drift during Diwali salesmedium
- Amount sent in paisemedium
- Fraud metrics once labels landmedium
- Reading a reliability diagrammedium
- Brier score by handmedium
- Proxy signals for fraudmedium
- Comparing a canary fairlymedium
- Predictions getting less certainmedium
- One bad minute pages on-callmedium
- Sampling logs to save costmedium
- Aadhaar numbers in logsmedium
- Measuring skew directlymedium
- MAPE for three forecastsmedium
- Residuals moving one waymedium
- Reading a PromQL latency querymedium
- Error budget for a 99.9% SLOmedium
- Profiling instead of storing rowsmedium
- SageMaker Model Monitor setupmedium
- Skew vs drift in Vertex AImedium
- When to roll back a modelmedium
- Scoring LLM answers at scalemedium
- Daily LLM billmedium
- More refusals after an updatemedium
- RAG answers without contextmedium
- Fraudsters change tacticsmedium
- Approval rates across groupsmedium
- Hour of day shifted by 5.5medium
- Monitoring a model on phonesmedium
- Replaying production inputsmedium
- Which input caused the score shiftmedium
- Gini from AUCmedium
- Precision moved, AUC did notmedium
- Launching in a new citymedium
- Drift jobs costing too muchmedium
- Counting distinct users cheaplymedium
- Monitoring with no labels at allmedium
- Real or virtual driftmedium
- Ensemble members disagree moremedium
- Which samples to label firstmedium
- Sampling for human reviewmedium
- How precise is 90% accuracymedium
- Newest month looks greatmedium
- NDCG for a short rankingmedium
- Hourly batch or streaming driftmedium
- Feature updates running latemedium
- Missing encoded two waysmedium
- Inputs far outside training rangemedium
- How useful are the alertsmedium
- More unknown tokensmedium
- Answers getting longermedium
- Spike in injection attemptsmedium
Hard 45
- Drift alerts firing all the timehard
- Model affects its own training datahard
- Drift check with a classifierhard
- Offline and online scores disagreehard
- High drift, stable performancehard
- Bug fixed offline onlyhard
- Every feature fails the testhard
- Chance alarms over 200 featureshard
- Drift on an unimportant featurehard
- Computing ECEhard
- Stable overall, bad in one cityhard
- Drift for free-text inputshard
- What MMD measureshard
- Confident on strange inputshard
- Catching a small steady shifthard
- How ADWIN workshard
- Mean and std did not movehard
- AUC steady, losses risinghard
- Timeouts turned into zeroshard
- user_id as a metric labelhard
- Weekly accuracy by the wrong datehard
- Detecting a smaller drophard
- Auto-retrain made it worsehard
- Judge prefers the first answerhard
- New embedding model, old indexhard
- Trusting thumbs-up ratingshard
- Recommender CTR falls after redesignhard
- Attributions shifted to one featurehard
- Labels changed, not usershard
- Combining quantiles across servershard
- Canary looked perfecthard
- Drift job silently stoppedhard
- Judge upgrade moved the scoreshard
- Alerts from tiny segmentshard
- Correcting for covariate shifthard
- Updating a score for a new base ratehard
- Estimating new class mix without labelshard
- Prediction sets cover lesshard
- Labels for a ±2 point marginhard
- Outcomes only for approved loanshard
- Setting alert thresholdshard
- Drift on 19,000 pincodeshard
- Comparing forecast errors fairlyhard
- Each feature fine, model nothard
- Last month’s accuracy changedhard