AI Engineer (LLM & Generative AI) interview questions and answers

RAG means we look things up before the model answers. When a user asks a question, we search our own knowledge base, usually by turning the question into an embedding and finding similar document chunks in a vector database, sometimes combined with keyword search. We take the best few chunks and put them into the prompt with instructions like "answer only using these sources and cite them". The model then writes an answer grounded in that context. We use it because the base model does not know our private documents and its knowledge has a cutoff date. RAG lets us use fresh and private data without retraining the model, reduces hallucinations, and lets users check citations. The quality of the answer depends heavily on the quality of retrieval, so we measure that separately.

Embeddings are lists of numbers that represent the meaning of text, images or other data. An embedding model is trained so that similar items end up close in vector space, and we compare them with cosine similarity or dot product. Besides RAG, I can use them for semantic search, where a query like "laptop will not charge" finds articles about "battery not charging". I can recommend similar products or articles. I can cluster support tickets to find common themes, and detect near-duplicate documents. Embeddings also make good features for simple classifiers: for example, a logistic regression on embeddings can route tickets to the right team cheaply. They are fast and cheap compared to calling a large model for every item, which makes them useful at scale.

LLMs are trained to predict the next likely token, so they produce fluent text even when they do not have the facts. If the answer is not in their training data or context, they may fill the gap with something that sounds right. In an application, the biggest fix is grounding: give the model the relevant facts through retrieval or tool calls, and tell it to answer only from those sources and cite them. I also explicitly allow "I do not know" so the model does not feel forced to guess. For factual tasks I use low temperature. Then I check outputs: for example verifying that cited passages actually contain the claim, validating numbers against the database, and using structured output so fields can be checked. Finally I measure hallucination rate on an eval set.

Most providers charge per token, with separate prices for input and output tokens, and output is usually more expensive. Speed also depends on tokens: the model processes the input first, which affects time to first token, then generates output one token at a time, so long answers take longer. In a chat app, history grows every turn, so cost per message grows unless I trim or summarise old messages. RAG adds retrieved chunks, so sending ten chunks when three are enough wastes money. To control this, I keep system prompts tight, retrieve fewer but better chunks, ask for concise answers or set max tokens, use prompt caching for repeated prefixes, and send simple tasks to smaller, cheaper models. I log token usage per feature so I can see where money goes.

Both settings control how random the output is. At each step the model gives a probability for every possible next token. Temperature rescales these probabilities: low temperature makes the top token much more likely, so output is focused and repeatable; high temperature flattens the distribution, so rarer tokens get picked and output is more varied. Top-p, or nucleus sampling, only samples from the smallest set of tokens whose probabilities add up to p, cutting off the unlikely tail. I usually change one, not both. For extraction, classification, SQL or code, I use low temperature for consistency. For brainstorming, marketing copy or story writing, I use a higher value for variety. Even at zero temperature, outputs may not be perfectly identical, so I still design evals and validation instead of relying on determinism.

A single LLM call takes a prompt and returns text once. An agent wraps the model in a loop and gives it tools, such as search, a database query or sending an email. On each turn the model either answers or asks for a tool call with arguments. My code runs the tool, appends the result to the conversation, and calls the model again, so it can plan, observe what happened and adjust. The loop stops when the model gives a final answer or when we hit a step or cost limit. Agents are powerful for open-ended tasks, but they are slower, more expensive and harder to test than a fixed workflow, so I only use one when the steps cannot be known in advance, and I trace every step.

Vector search finds text by meaning rather than exact words. First I split documents into chunks and run each through an embedding model, which turns it into a list of numbers where similar meanings end up close together. These vectors go into an index, often a vector database. At query time I embed the question with the same model and ask the index for the nearest vectors, usually by cosine similarity or dot product. Approximate nearest-neighbour structures such as HNSW make this fast even with millions of chunks. It is useful because a question like "how do I get my money back" can match a section titled "refund policy". Its weakness is exact terms like error codes or names, so I often combine it with keyword search.

Rate-limit errors usually come back as HTTP 429. First I make the client handle them properly: read the Retry-After header if present, otherwise retry with exponential backoff plus random jitter, and stop after a few attempts so users are not left waiting forever. Then I look at why we are hitting the limit. Limits are often on both requests and tokens per minute, so trimming long prompts or capping output length can help as much as slowing down. I put a queue or semaphore in front of the API to control concurrency, and move non-urgent work to batch jobs. Finally I add monitoring on usage against limits, ask the provider for a higher tier if needed, and consider a fallback model for peak times.

Zero-shot prompting gives the model only instructions, like "classify this ticket as billing, technical or other". Few-shot prompting adds a handful of worked examples of input and correct output. I start zero-shot because strong models follow clear instructions well and it keeps prompts short and cheap. I add examples when the output format is unusual, when the boundary between categories is subtle, or when I need a specific tone, because examples often show these better than rules. The examples must be varied and representative; if they all have the same label or length, the model copies that pattern. Since examples add tokens to every call, I compare both versions on a small eval set and keep the cheaper one if quality is equal.

I start with data minimisation: only send the fields the task actually needs. If the summary does not need card numbers or phone numbers, I redact or mask them before the call, often with a PII detection step. I check the provider terms for retention and whether data is used for training, and pick settings or regions that match our privacy rules and contracts. Logs and traces are an easy place to leak data, so I mask or limit what is stored and set retention periods. In RAG features I enforce access controls in retrieval, so a user can never get another person record into the prompt. Prompt instructions like "do not reveal" are not protection; real controls must sit in code.

I start with prompting and build an eval set first, so I can measure every option. Good instructions and a few examples solve many problems cheaply and quickly. If the gap is missing knowledge, such as company documents, product data or anything that changes often, I add RAG or tool calls, because fine-tuning is a poor way to store facts and is hard to update. If the gap is behaviour, like a strict output format, a domain writing style, or a classification task the model keeps getting wrong despite good prompts, fine-tuning helps, especially when I have hundreds or thousands of good examples. Fine-tuning a smaller model can also cut cost and latency once a large model has proven the task. These approaches combine: a fine-tuned model can still use RAG. I always compare against the prompted baseline.

Before launch I build an eval set of real or realistic inputs, including common cases, edge cases and adversarial ones, with expected outcomes or grading criteria. I use the cheapest reliable grader for each check: code checks for format, exact values and JSON validity; an LLM judge with a clear rubric for qualities like helpfulness and faithfulness to sources; and human review on a sample, which I also use to calibrate the judge. For RAG I evaluate retrieval separately with recall at k, then generation for correctness and citation support. I run the suite on every prompt, model or pipeline change and compare per category to catch regressions. After launch I track thumbs up and down, task completion, escalation rate, latency and cost, and regularly add failed production cases back into the eval set.

First I measure. I collect queries where the answer is known to be in the corpus, label the correct chunks, and compute recall at k. Then I look at failures. Often the problem is parsing: tables, PDFs or scanned pages extracted badly. Next is chunking: I split along headings, add overlap, and prepend document and section titles so chunks have context. If users search for codes, names or exact terms, I add keyword search like BM25 and fuse it with vector results. Query rewriting helps when user questions are short or conversational, for example turning "what about refunds?" into a full question using chat history. Retrieving more candidates and using a cross-encoder reranker improves precision. Finally, I test a different embedding model on my labelled set. I change one thing at a time and track recall.

I define a clear schema: field names, types, enums for things like currency, and nullable fields so the model can say a value is missing instead of inventing it. I use the provider structured output or tool calling so responses follow the schema. Then I validate in code with something like Pydantic or Zod, and add business checks: line items should add up to the total, dates should be valid, the tax ID should match a pattern. If validation fails, I retry with the error message included, and if it still fails or values look suspicious, I send the invoice to a human review queue. Low temperature helps consistency. Before launch I build a labelled set of real invoices and measure accuracy per field, and I keep monitoring correction rates from reviewers after launch.

The model only knows what the tool name, description and schema tell it, so I write them like documentation for a new developer: what the tool does, when to use it, and what each parameter means, with types and enums. I prefer a small set of focused tools over many overlapping ones, because overlap confuses the model. Tool results should be concise and relevant, not huge raw dumps that waste context, and errors should explain what went wrong so the model can fix its call. On the application side I validate every argument, enforce the user permissions in code rather than trusting the model, and follow least privilege. For actions with side effects, like refunds or sending emails, I make them idempotent and require confirmation. I test tool selection with evals that check the right tool and arguments are chosen.

I split the evaluation in two. For retrieval I build a labelled set of real queries, each mapped to the chunks or documents that contain the answer, often with help from domain experts or by generating questions from known passages and reviewing them. Then I compute recall@k, because the model cannot use a chunk that was never retrieved, and MRR or nDCG to see whether good chunks rank near the top. Precision@k tells me how much noise goes into the prompt. For generation I separately check faithfulness to the retrieved context and answer correctness, using rubric-based judges calibrated on human labels. Keeping them apart tells me where to fix things: if recall is low I work on chunking, embeddings, hybrid search or reranking; if recall is fine I work on the prompt or model.

The browser calls my backend, never the provider directly, so the key stays secret. The backend opens a streaming request to the model and forwards text deltas to the client, usually as server-sent events, and the UI appends them as they arrive. This mainly cuts time to first token, which is what users feel. There are several details to handle. Tool-call arguments stream as JSON fragments, so I buffer them per call and parse only when complete. If the user closes the tab, I cancel the upstream request to stop paying for tokens. Errors can happen mid-stream, so the UI needs a clear failure state and retry. Since text is shown before it is finished, any structured output must be validated at the end, and for sensitive apps I run moderation in chunks.

I think in three layers. First, provider prompt caching: if long system prompts, tool definitions or documents are a stable prefix, the provider reuses that work, cutting cost and latency, so I put static content first and changing content last. Second, exact-match caching of full responses keyed by normalised input, model and prompt version. It is safe but only hits for repeated questions. Third, semantic caching, which returns a stored answer for a similar question. It raises hit rates but can return wrong answers when phrasing is close but meaning differs, so I use a strict threshold, measure false hits on labelled pairs, and skip it for actions. For all of them, the cache key must include user or tenant and permissions so nobody gets someone else data, and entries need expiry when sources change.

I instrument every request as a trace, with spans for each model call, retrieval step and tool call, often using OpenTelemetry-style conventions so it works with existing tooling. Each span records inputs and outputs, prompt version, model, token counts, cost, latency and errors. That lets me debug a single bad answer and also see trends. On top I add product signals such as thumbs up or down, edits and retries, and run automated online evals, like groundedness or format checks, on a sample of traffic. Dashboards and alerts cover error rates, time to first token, p95 latency, cost per request and drops in quality scores, broken down by prompt version so I can tie regressions to deploys. Because traces hold user text, I mask PII and set strict retention and access.

I start from real inputs, sampled from logs or pilot users, so the set reflects actual traffic rather than what I imagine. I make sure it covers the main intents plus edge cases: empty input, long input, other languages, and questions we should refuse. For each case I write an expected answer or a grading rubric together with domain experts, since some tasks have many valid answers. Every bug report becomes a new case, so past failures never quietly return. The set is versioned in the repo and runs in CI on every prompt, model or retrieval change, with results compared per category. I keep it large enough to detect the size of change we care about, and refresh it regularly from production so it follows shifts in what users ask.

I would start with ingestion: connectors pull documents from each source, parse them carefully (including PDFs and tables), split them along structure, and store chunks with metadata such as source, date, owner and access control list. Updates are incremental using change events, and deletions remove vectors quickly. Retrieval uses hybrid search, vector plus BM25, and always applies a permission filter based on the user identity at query time, so the model never sees chunks the user cannot access; filtering after generation is too late. A reranker narrows to the top few chunks. The generator is instructed to answer only from sources, cite them, and say when it cannot find an answer. I build an eval set per department, measure retrieval recall and answer faithfulness, and monitor latency, cost per query, feedback and stale content.

I assume prompt injection will happen, because any email can contain instructions written for the model. So the design must stay safe even if the model is fooled. First, untrusted content is clearly separated from instructions and treated as data. Second, least privilege: the agent only gets tools it needs, with scoped credentials, for example read-only access by default and sending only to allowlisted domains. Third, sensitive actions such as sending email, deleting data or making payments go through code-level policy checks and user confirmation that shows exactly what will happen. Fourth, I block common exfiltration routes, like rendering external image links containing data, and scan outputs for secrets and personal data. Finally I log every tool call, monitor for unusual patterns, and run regular red-team tests with known injection attacks in my eval suite.

First I add tracing so I can see each step: the prompt, the tool calls, the results and where it went wrong. Usually failures cluster: wrong tool choice, bad arguments, confusing tool output, or losing track of the goal. Then I simplify. Many tasks do not need a free-form agent; a fixed workflow with LLM steps at specific points is more predictable, so I only keep autonomy where it adds value. I improve tool descriptions and errors, keep explicit state like a task plan or checklist, and validate each step output. I add hard limits on steps, tokens and time, retries for transient tool errors, and checkpoints so long tasks can resume. Risky actions get human approval. Finally I build task-level evals from real production failures and measure success rate, cost and steps per task on every change.

I start by measuring where time and money go: input and output tokens per request, number of model calls, retrieval time and time to first token. Often there are quick wins: a bloated system prompt, too many retrieved chunks, or several sequential calls that could run in parallel or be merged. I reduce output length with clear instructions and max tokens, since output tokens are slow and costly. I add model routing, where a small fast model handles simple requests and only hard ones go to the large model; sometimes a fine-tuned small model replaces the large one for a narrow task. Prompt caching cuts cost for repeated prefixes, and caching full responses helps for repeated questions. Streaming improves perceived latency. For offline jobs I use batch APIs. Every change is checked against the eval set so quality does not slip.

I consider it when the task is narrow and high volume, such as classifying tickets or extracting fields, and a large model already does it well but is too expensive or slow at scale. Data privacy or on-premise rules are another strong reason. The steps: define the task and an eval set first. Collect training data from labelled history or by having the large model produce outputs that humans spot-check, which is a form of distillation; check the provider terms allow that. Clean the data carefully, since quality matters more than size. Fine-tune with a parameter-efficient method like LoRA, which is cheap and keeps the base model frozen. Compare the fine-tuned model against the large model on held-out data, including edge cases. Finally plan serving with something like vLLM, monitoring, and a process to retrain when data drifts.

I start with memory. Weights need parameters times bytes per parameter, so a model in 16-bit needs about 2 bytes per parameter, and 8-bit or 4-bit quantization cuts that but must be checked on our evals. Then the KV cache: per token it is 2 × layers × KV heads × head size × bytes, multiplied by context length and concurrent sequences, and it often dominates at long contexts. I would use a serving engine with continuous batching and paged KV cache, which raises throughput a lot. The key trade-off is throughput versus latency: bigger batches lower cost per token but raise time per token, so I set limits from our latency targets. Prefix caching and speculative decoding can help further. Finally I load test with realistic prompt and output lengths, and autoscale on queue depth.

First I confirm and size the problem with data: feedback rates, retries, online eval scores and abstentions, split by intent, language and customer, to see whether it is broad or one segment. Then I list everything that can change silently: prompt versions, the provider model behind an alias, the embedding model, the index contents, tool APIs, and upstream data. I pull traces of bad examples and walk through each span to find the failing step, for example retrieval returning stale chunks or a tool now returning a new format. I also compare this week inputs with last week to spot drift, such as new topics after a launch. Once I find the cause, I fix it, rerun the golden set, add the failing cases to it, and add alerts or pinned versions so it is caught sooner next time.

I make the judging task narrow. Instead of "rate quality 1 to 10", I ask specific yes or no questions, such as "is every claim supported by the context?" or "does it follow the refund policy?", each with a clear rubric and examples. I give the judge the reference answer or source documents when I have them. Then I calibrate: experts label a few hundred examples with enough failures in them, and I measure the judge recall on bad answers and a chance-corrected agreement like Cohen kappa, not raw accuracy, which looks good under class imbalance. I counter known biases by swapping order in pairwise tests, penalising padding, and avoiding the same model judging its own outputs. Whenever the judge model or rubric changes, I recalibrate, and I keep sampling cases for human review.

I start with the simplest option that works. If the steps are known, such as classify, retrieve, draft, check, a fixed workflow is cheaper, faster and easier to test. If the path depends on what the model finds, a single agent with a well-designed tool set handles most cases. I move to multiple agents only for a clear reason. One is parallelism: an orchestrator can send several subagents to research different parts at once. Another is context isolation: each subagent works in its own fresh context and returns a short summary, which keeps the main context clean on long tasks. The costs are real: many more tokens, more latency, lost information at handoffs, and harder debugging. So I compare designs on the same eval set and use tracing across agents before committing.

I treat every MCP server as untrusted code and its output as untrusted input. Tool descriptions and results go straight into the model context, so a malicious server can inject instructions. I review descriptions before enabling a server, pin versions so they cannot change silently, and run local servers in a sandbox with limited file and network access. Each server gets its own scoped credentials with least privilege, and OAuth tokens are never passed between servers. I watch for the dangerous mix of private data, untrusted content and a way to send data out, because together they allow exfiltration; I remove one leg or require human approval for actions like sending email, payments or deleting data. Finally I log every tool call with its arguments for audit, and red-team the setup with injected content.