Shadow deployment or canary rollout for this model — and what would you measure in each?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A search team has a new ranking model that is better on offline metrics (illustrative). The prior model is in production serving all traffic. The new model uses two new features from a service that has never taken production load. Business asks for it to be live by the end of the quarter.
What it is really testing
Whether the candidate knows the difference — shadow serves no user and canary serves a few — and can match each to what it can and cannot tell you: shadow measures latency, errors, feature availability and agreement, but not outcomes; canary measures outcomes on a slice, at the cost of exposing users. The two new features from an untested service are the clue.