41. Shipping a prompt change safely
A teammate rewrote the system prompt to fix one customer complaint. How should you decide whether to ship it?
- A.Run the full eval set (real and edge-case examples with expected outcomes) before and after, compare scores per category, then ship gradually while monitoring
- B.Test the one complaint example manually; if it works, ship it
- C.Ship it and wait for complaints
- D.Ask the model whether the new prompt is better
Show answer
Answer: A. Run the full eval set (real and edge-case examples with expected outcomes) before and after, compare scores per category, then ship gradually while monitoring
Prompt changes often fix one case and break others. A versioned eval set with clear pass criteria, run automatically like unit tests, catches regressions. Looking at results by category and rolling out gradually with production monitoring makes the change safe.
42. Prompting a reasoning model
You move a complex analysis task to a model with built-in extended reasoning. Your old prompt had a very detailed, rigid list of 15 thinking steps. What is generally the better approach?
- A.Make the thinking steps even more detailed
- B.Remove all context and constraints
- C.Give a clear goal, the relevant context, constraints and success criteria, and let the model plan its own reasoning; then check results with evals
- D.Ask the model to answer immediately without thinking
Show answer
Answer: C. Give a clear goal, the relevant context, constraints and success criteria, and let the model plan its own reasoning; then check results with evals
Models with built-in reasoning often do better with high-level guidance than with micromanaged steps, which can limit their approach. What they still need is a clear objective, the right information and how the output will be judged. As always, compare both prompt styles on your eval set.
43. Invented values in extraction
An extraction prompt returns "phone_number" for every resume, even those without one, sometimes inventing numbers. What is the best fix?
- A.Define the schema with optional fields, instruct the model to return null when the value is not present in the text, include an example with nulls, and validate values against the source
- B.Tell the model the phone number is very important
- C.Remove phone_number from the schema permanently
- D.Raise temperature so the model explores more options
Show answer
Answer: A. Define the schema with optional fields, instruct the model to return null when the value is not present in the text, include an example with nulls, and validate values against the source
When every field looks required, models tend to fill it. Explicit permission and an example for null values, plus a schema marking fields optional, reduce invention. A code check that the extracted value really appears in the source text catches the rest.
44. Prompt cache hit rate near zero
Your long system prompt should be cached, but the cache hit rate is almost zero. Look at how the prompt is built. Why?
system = f"""Current time: {datetime.now().isoformat()}
You are the support assistant for Acme...
(6,000 tokens of fixed policy and tool docs)"""- A.Caching only works for prompts under 1,000 tokens
- B.The policy text is too long to cache
- C.Python f-strings cannot be cached
- D.The timestamp changes on every request and sits at the very start, so the prefix is never identical; move dynamic values to the end
Show answer
Answer: D. The timestamp changes on every request and sits at the very start, so the prefix is never identical; move dynamic values to the end
Prefix caching needs the leading tokens to match exactly. A value that changes each call at the top makes every request a cache miss for everything after it. Put the time, user data and other variables after the stable block, or round the time if it must appear early.
45. A more honest confidence score
Your classifier asks the model to also print "confidence: 0–100". The numbers are always 85–95, even on wrong answers. What is a better signal of confidence if the API supports it?
- A.The token log probabilities for the label the model output
- B.The length of the model's reply in characters
- C.The temperature value used for the call
- D.Asking the model to explain why it is confident
Show answer
Answer: A. The token log probabilities for the label the model output
Self-reported confidence is just more generated text and is often poorly calibrated. Log probabilities show how much probability the model actually gave to the chosen label token versus alternatives, which tends to track correctness better. You should still check calibration on labelled data.
46. Solving easy sub-problems first
A prompt first asks the model to list the sub-questions needed to solve a tricky problem, then solves them in order, feeding each answer into the next. What is this approach and when does it help most?
- A.Self-consistency; when the answer is a single word
- B.Least-to-most prompting; when later steps need earlier answers
- C.Prefilling; when output must start with a fixed token
- D.Zero-shot prompting; when the task has no steps
Show answer
Answer: B. Least-to-most prompting; when later steps need earlier answers
Least-to-most prompting decomposes a problem into simpler sub-problems and solves them in sequence, each using the previous results. It helps on compositional tasks harder than any example shown, where one-pass reasoning tends to skip steps.
47. Exploring several reasoning paths
What distinguishes tree-of-thoughts prompting from plain chain of thought?
- A.It writes one line of reasoning and then translates it
- B.It explores and scores several reasoning paths, and can backtrack
- C.It removes reasoning and asks only for the final answer
- D.It shares reasoning between different users' requests
Show answer
Answer: B. It explores and scores several reasoning paths, and can backtrack
Chain of thought follows one path. Tree of thoughts generates multiple candidate steps, evaluates which look promising, and searches (for example breadth-first) with the option to drop dead ends. It costs many more calls but helps on puzzles and planning.
48. A strict 160-character limit
SMS alerts must be at most 160 characters. The prompt says so, yet about 8% of outputs are 170–190 characters. What is the most reliable solution?
- A.Repeat the 160-character rule five times
- B.Use temperature 0 so the model counts exactly
- C.Ask the model to count characters out loud first
- D.Check length in code and regenerate or shorten when it is over
Show answer
Answer: D. Check length in code and regenerate or shorten when it is over
Models see tokens, not characters, so they estimate length poorly. Keep the instruction (and aim a bit lower), but enforce hard limits in code: measure, then retry with feedback or trim. Never rely on the prompt alone for a hard constraint.
49. Examples with random labels
A 2022 study ("Rethinking the Role of Demonstrations") found that replacing the correct labels in few-shot examples with random labels often hurt classification accuracy only slightly. What does this suggest?
- A.Examples teach format and label set, not only the mapping
- B.Labels in examples are always ignored, so remove them
- C.Wrong labels improve accuracy and should be preferred
- D.Few-shot prompting has no effect on model output
Show answer
Answer: A. Examples teach format and label set, not only the mapping
The study showed that much of the benefit comes from showing the label space, the kind of inputs and the output format. That does not mean labels never matter; on hard or unusual tasks, correct labels still help, so keep them right.
50. Banning a word with logit_bias
You try to stop the word "guarantee" by setting logit_bias to -100 for the token ID of "guarantee". The word still appears sometimes. Why?
- A.The word has other token forms, like " guarantee"
- B.logit_bias only works when temperature is 2
- C.A bias of -100 makes the token more likely
- D.logit_bias applies only to the system prompt
Show answer
Answer: A. The word has other token forms, like " guarantee"
Logit bias works on token IDs, and one word can map to several tokens: " guarantee", "Guarantee", or pieces like "guar" + "antee". Banning one ID misses the variants, and banning shared pieces can break other words. Post-checks in code are often more reliable.
51. Old and new policies both retrieved
A HR bot retrieves both the 2024 and 2026 leave policies, which disagree, and sometimes answers from the old one. What is the best prompt-side fix?
- A.Ask the model to average the two policies
- B.Tell the model to pick whichever policy is longer
- C.Remove all dates so the model is not confused
- D.Show each document's date and say to follow the latest
Show answer
Answer: D. Show each document's date and say to follow the latest
Without dates, the model cannot tell which source is current. Including metadata such as effective date and telling it to prefer the most recent valid policy resolves conflicts; better still, filter superseded documents out at retrieval time.
52. Twenty reviews in one call
To save cost, you send 20 reviews per call and ask for 20 sentiment labels. Sometimes labels shift by one position or one review is skipped. What is the best fix?
- A.Send 40 reviews per call so errors average out
- B.Ask for labels in reverse order
- C.Remove line breaks between reviews to save tokens
- D.Give each review an ID and require output keyed by those IDs
Show answer
Answer: D. Give each review an ID and require output keyed by those IDs
When outputs are matched by position, one skip misaligns everything after it. Numbering items and asking for {id: label} output lets code detect missing or extra IDs and retry just those, while keeping the batching savings.
53. Cost of a small-then-large cascade
Every request goes first to a small model costing ₹0.10. In 30% of cases its answer fails a check and the request is sent again to a large model costing ₹1.00. What is the average cost per request?
- A.₹0.40
- B.₹0.37
- C.₹0.30
- D.₹1.10
Show answer
Answer: A. ₹0.40
Everyone pays the small call: ₹0.10. Then 30% also pay ₹1.00, which adds 0.3 × 1.00 = ₹0.30. Average = ₹0.40, versus ₹1.00 if every request went to the large model. The cascade only pays off if the check is reliable.
54. Per-request tool lists and caching
With Anthropic prompt caching, your app adds or removes tool definitions on each request depending on the user's plan. Cache hits are rare. Why?
- A.Caching only works when there are no tools at all
- B.Tools are cached separately and never expire
- C.The cache only stores the last user message
- D.Tools start the cached prefix, so changes invalidate it
Show answer
Answer: D. Tools start the cached prefix, so changes invalidate it
The cached prefix is built in order: tools, then system, then messages. Any change to tool definitions changes the very start of the prefix, so nothing after it can be reused. Keep the tool set stable, or group users into a few fixed tool sets.
55. An LLM that favours option A
You use an LLM to answer multiple-choice questions and notice it picks the first option more often than chance. What is a robust mitigation?
- A.Always put the correct answer in position A
- B.Ask several times with shuffled options and take the majority
- C.Remove the option letters and ask for free text only
- D.Use a higher temperature for every question
Show answer
Answer: B. Ask several times with shuffled options and take the majority
Position bias means the order of options changes the answer. Permuting options across runs and aggregating cancels the bias; it also tells you which items are unstable.
56. Tips and threats in prompts
A blog claims that adding "I will tip you $200 for a perfect answer" improves quality by 10%. What is the right response for a production prompt?
- A.Add it immediately since it is free
- B.Test it on your own eval set; effects vary a lot
- C.Add a threat as well to double the claimed effect
- D.Assume it works because the blog had charts
Show answer
Answer: B. Test it on your own eval set; effects vary a lot
Reported effects of emotional or reward phrases are inconsistent and often vanish on newer models or other tasks. Only a controlled comparison on your own data shows whether it helps; clear instructions are usually the better lever.
57. Optimising prompts from data
A framework lets you declare a task's inputs and outputs, provide a metric and some labelled examples, and then automatically searches for instructions and few-shot examples that maximise the metric. Which tool is best known for this?
- A.DSPy
- B.LangSmith
- C.Ollama
- D.Pydantic
Show answer
Answer: A. DSPy
DSPy treats prompts as parameters of a program and uses optimisers to choose instructions and demonstrations against a metric. It needs a good metric and data; otherwise it can overfit to a small set.
58. Forcing a PAN number format
You extract Indian PAN numbers, which have the pattern five capital letters, four digits, one capital letter. With an open-weight model, how can you guarantee the output matches this pattern?
- A.Ask the model nicely to follow the pattern
- B.Set temperature to 0 so it never makes mistakes
- C.Use regex-constrained decoding: [A-Z]{5}[0-9]{4}[A-Z]
- D.Add ten examples of PAN numbers to the prompt
Show answer
Answer: C. Use regex-constrained decoding: [A-Z]{5}[0-9]{4}[A-Z]
Constrained or grammar-guided decoding (for example with libraries like Outlines or a server's regex/grammar option) masks any token that would break the pattern, so output always matches. It guarantees format, not that the value is correct, so still verify against the document.
59. Where to put tool usage rules
An agent has 15 tools. Usage rules for each tool are in a long system prompt section, and the model often misuses them. What usually helps?
- A.Delete all tool descriptions to shorten the prompt
- B.Merge all 15 tools into one generic tool
- C.Repeat the system prompt section after every message
- D.Move each tool's usage guidance into its own description
Show answer
Answer: D. Move each tool's usage guidance into its own description
The model reads a tool's description at the moment it decides whether and how to call it. Putting when-to-use, when-not-to-use and argument rules there keeps guidance next to the tool and reduces confusion between similar tools.
60. How many tokens min_p keeps
Next-token probabilities are 0.50, 0.20, 0.08, 0.05 and 0.04 (others smaller). With min_p = 0.1, a token is kept if its probability is at least min_p × the top probability. How many of these five tokens are kept?
- A.4
- B.2
- C.3
- D.5
Show answer
Answer: A. 4
Threshold = 0.1 × 0.50 = 0.05. Tokens with 0.50, 0.20, 0.08 and 0.05 meet it; 0.04 does not. So 4 are kept. min_p adapts: when the top token is very likely, the threshold rises and fewer options survive.
