You are changing the embedding model. What kind of change is that?
Whether the candidate recognises a data migration disguised as a configuration change, and whether they know what makes vectors from two models incomparable.
The situation behind the question
Interviewers ask this because it happened to them.
A retrieval index backs an in-product assistant. A better embedding model is available. The proposal is to update the configuration and let the index rebuild over the weekend.
A strong answer
Flags
Green flags
- Recognises re-embedding as a migration with a cutover, a validation and a rollback.
- Knows vectors from different models are not comparable, and can say why a mixed index is worse than either.
- Wants a versioned evaluation set so quality is measured rather than felt.
- Records embedding version, chunk strategy and source version as data rather than as configuration.
Red flags
- Treats it as a configuration change with a rebuild.
- Plans an in-place rebuild with no account of what the index contains partway through.
- Has no way to evaluate whether the new model is better for this corpus.
- Believes rolling back the model version repairs an index that already holds mixed vectors.
Follow-ups
Where the conversation goes if the first answer holds up.
- The chunk strategy is changing too. Does anything about your plan get harder?
- How would you detect a retrieval quality regression in production, given that no monitor fires?
- The corpus is large enough that a full re-embed takes days. How do you sequence the cutover?