Prompt Engineer
LLM evaluation questions
149 questions on LLM evaluation for prompt engineers, easiest first.
Easy 42
- Reducing made-up answerseasy
- Testing a prompt properlyeasy
- What a hallucination iseasy
- What a golden set iseasy
- Quote first, then answereasy
- What exact match measureseasy
- Accuracy from four countseasy
- What a false positive iseasy
- 99% accuracy that is uselesseasy
- Code checks or an LLM judgeeasy
- Tracking broken JSONeasy
- Grading with or without a referenceeasy
- Offline and online evaluationeasy
- Relying on thumbs-up ratingseasy
- Why keep a regression suiteeasy
- Hiding which model wrote whateasy
- What p95 latency meanseasy
- A test set unlike real trafficeasy
- Comparing scores across monthseasy
- First step in evaluating a new boteasy
- What perplexity tells youeasy
- A top benchmark scoreeasy
- How HumanEval scores codeeasy
- A library for RAG metricseasy
- A business metric for a support boteasy
- What TTFT measureseasy
- Measuring needless refusalseasy
- Before hiring human graderseasy
- Temperature for an LLM judgeeasy
- Pass/fail per criterioneasy
- What faithfulness checks in RAGeasy
- What answer relevance checkseasy
- Computing retrieval hit rateeasy
- Computing precision at 5easy
- Budget for one eval runeasy
- Recall of a document findereasy
- How big a first eval seteasy
- Same question, different wordseasy
- Which prompt produced this replyeasy
- When the answer key is wrongeasy
- Replies slowly getting longereasy
- Evals compared with unit testseasy
Medium 55
- Judging summary qualitymedium
- The judge prefers answer Amedium
- A small prompt fix shipsmedium
- Which metric for auto-refundsmedium
- Vague 1-10 judge scoresmedium
- Computing F1medium
- Partial credit with token F1medium
- Macro or micro F1medium
- Agreement beyond chancemedium
- High ROUGE, wrong summarymedium
- Low BLEU for a good translationmedium
- Cosine similarity as a gradermedium
- Order of judge outputmedium
- Making judge scores consistentmedium
- Teaching the judge with examplesmedium
- Win rate when there are tiesmedium
- Which traces to reviewmedium
- Turning reviews into prioritiesmedium
- Generating varied synthetic testsmedium
- Testing the awkward inputsmedium
- Evals in the deploy pipelinemedium
- Reading a promptfoo testmedium
- Testing whole conversationsmedium
- Grading an agent's outcomemedium
- Measuring tool callingmedium
- Extraction scored too harshlymedium
- Picking a judge modelmedium
- Judging factual correctnessmedium
- A cheaper groundedness checkmedium
- Measuring hallucinations more finelymedium
- Two sides of a summarymedium
- Graders disagree on 15% of itemsmedium
- Kappa of 0.25 among expertsmedium
- Judge and humans ranking modelsmedium
- Signals beyond ratingsmedium
- Rolling out a model changemedium
- Scores fine, complaints risingmedium
- Finding which labels get mixed upmedium
- Building tests from real chatsmedium
- Two kinds of eval suitesmedium
- Open-source LLM observabilitymedium
- An eval framework from a safety institutemedium
- Safer but less helpfulmedium
- Absolute or comparative judgingmedium
- Model names in the judge promptmedium
- Flagging chats for human reviewmedium
- Evaluating citationsmedium
- Judging Arabic answersmedium
- A great answer with one dangerous errormedium
- A cheaper model that is almost as goodmedium
- Tools for human labellingmedium
- Splitting logs into dev and testmedium
- Grading 20 answers in one judge callmedium
- Pairwise judging for maths answersmedium
- Choosing a confidence thresholdmedium
Hard 52
- Perfect on 20 exampleshard
- Can we trust the LLM judgehard
- Switching to a newer modelhard
- Invented values in extractionhard
- Judge and candidate from one familyhard
- A suspicious score jumphard
- Is a 2-point gain real?hard
- Where is the RAG bug?hard
- Real sources, wrong claimshard
- Longer answers keep winninghard
- Tiny wording, big swinghard
- Errors flowing down a chainhard
- Scores change between runshard
- The model agrees with everythinghard
- Evals green, users unhappyhard
- Adjusting for an imperfect judgehard
- Margin of error for a pass ratehard
- How many cases for ±2 pointshard
- Comparing two prompts on the same caseshard
- Computing pass@2hard
- Succeeding every time, not oncehard
- Better overall, worse for Arabichard
- Answers that praise themselveshard
- Bootstrap for a score differencehard
- Examples taken from the test sethard
- Is the model's confidence honesthard
- Ranking many prompts from pairshard
- Two retrieval metrics for RAGhard
- Testing on live traffic without riskhard
- An A/B test that looks too goodhard
- How well the judge finds failureshard
- Scoring extracted line itemshard
- Average precision for one queryhard
- Twenty variants, one "winner"hard
- An eval set full of near-copieshard
- Scores jump with no prompt changehard
- What the judging costshard
- Correct answers marked wronghard
- Every variant scores 98%hard
- A simulated user that is too nicehard
- Agent results vary by run orderhard
- Does the offline score matterhard
- Relevance labels from an LLMhard
- Faithful to a wrong documenthard
- Is the variance from the judgehard
- Auto-approving refunds safelyhard
- No failures seen in 150 testshard
- Comparing models by error costhard
- Editing the judge mid-quarterhard
- Passing the needle testhard
- A 30-case Tamil slicehard
- Eval mix differs from traffichard