medium10 pts · first answer countsmonitoringdrift

Users start asking new things

After a new product launch, your RAG assistant gets more "I do not know" answers, but offline eval scores on the golden set are unchanged. What is the most likely explanation and fix?

A.The hosted model weights drifted over time, so fine-tune it againB.The golden set is too large to be sensitive, so cut it in halfC.Temperature is too low for new questions, so raise it to around 1D.Queries drifted to new topics; track retrieval misses, add docs, refresh the eval set

← Running SQL that an LLM wrote Take the medium mock interview →