Your RAG system has 200 million chunks indexed with an old embedding model. How do you move to a new embedding model without downtime or quality loss?

What the interviewer wants to hear

• Old and new vectors are not compatible, so a full re-embed is needed
• Build the new index in parallel, with dual writes for new documents
• Evaluate retrieval quality on a labelled query set before switching
• Switch with an alias or flag, shadow or A/B test, and keep rollback
• Plan cost and throughput of the backfill

Model answer

Vectors from different embedding models live in different spaces, so I cannot mix them; the whole corpus must be re-embedded. I would create a new index alongside the old one and start dual writes, so new and updated documents go to both. Then I run a backfill job over the 200 million chunks, batched for GPU throughput, idempotent and resumable, with rate limits so it does not hurt production. I estimate the cost first, since this can be large. Before switching, I evaluate retrieval on a labelled query set, comparing recall at k and answer quality, and tune index settings for the new dimension. I then shadow or A/B test the new index on live queries. The switch itself is a config flag or index alias, so rollback is instant, and I keep the old index until the new one has proven itself.

More hard MLOps Engineer questions

• Design an internal ML platform for 15 data science teams who currently deploy models in different ad hoc ways.
• How would you self-host and serve an open-weight LLM for thousands of concurrent users?
• Your recommendation model suddenly shows a 25% drop in click-through rate. Walk through your incident response.
• The ML infrastructure bill doubled in six months. How do you bring it down without slowing teams?
• A regulator asks you to show exactly how a credit decision model was built and how a past decision was made. How should your MLOps setup support this?
• You need to run a multi-day, multi-node training job on preemptible GPUs. How do you make it reliable and cost-effective?
• Users say a self-hosted LLM feature is too slow. How do you reduce latency without destroying throughput or quality?
• Under privacy law, a customer asks you to delete all of their data. How should an ML platform handle this across datasets, features, indexes and models?
• How do you build a real-time feature pipeline whose values match the batch features used in training?