How does vector search work, and why is it useful for an AI application?

What the interviewer wants to hear

• Text is turned into embeddings that capture meaning
• Documents are embedded once and stored in an index
• The query is embedded with the same model and compared by similarity
• Approximate nearest-neighbour indexes make search fast at scale
• It finds matches by meaning, but can miss exact terms like codes

Model answer

Vector search finds text by meaning rather than exact words. First I split documents into chunks and run each through an embedding model, which turns it into a list of numbers where similar meanings end up close together. These vectors go into an index, often a vector database. At query time I embed the question with the same model and ask the index for the nearest vectors, usually by cosine similarity or dot product. Approximate nearest-neighbour structures such as HNSW make this fast even with millions of chunks. It is useful because a question like "how do I get my money back" can match a section titled "refund policy". Its weakness is exact terms like error codes or names, so I often combine it with keyword search.

More easy AI Engineer (LLM & Generative AI) questions

• Explain retrieval-augmented generation (RAG) to a new team member. Why do we use it?
• What are embeddings, and what can you build with them besides RAG?
• Why do LLMs hallucinate, and what simple steps reduce it in an application?
• How do tokens affect the cost and speed of an LLM application?
• What do temperature and top-p control, and how would you set them for different tasks?
• What is an LLM agent, and how is it different from a single LLM call?
• Your app starts getting rate-limit errors from the LLM API. How do you handle them?
• What is the difference between zero-shot and few-shot prompting, and when would you use each?
• How do you protect personal data when building a feature on a hosted LLM API?