hard20 pts · first answer countsdeployment-strategiesexperimentation

Canary that cannot catch quality drops

Your canary sends 1% of traffic to a new model for 30 minutes and checks conversion rate. A model that lowered conversion by 3% passed. Why?

A.Canaries should use 0% of trafficB.Conversion rate cannot be measuredC.At that traffic and duration the canary has far too little statistical power to detect a 3% changeD.The model was rolled back too early

Discussion 0

Join free to post in the discussion.

Canary that cannot catch quality drops

Discussion 0

Related questions