LLM evaluation
39 practice items · needed by 4 roles
Easy 9Medium 9Hard 21
Practice questions
- LLM judge prefers the first answerhard
- Why keep a separate test seteasy
- Computing precisioneasy
- What a golden set is foreasy
- Pairwise LLM judge biasmedium
- Computing recall at 5medium
- Computing MRRmedium
- Judge prefers longer answersmedium
- Shipping a prompt change safelyhard
- A judge with 90% agreementhard
- Computing nDCG at 3hard
- Is a 4-point gain realhard
- Fine-tuned model scores too wellhard
- Reducing made-up answerseasy
- Testing a prompt properlyeasy
- What a hallucination iseasy
- What a golden set iseasy
- Quote first, then answereasy
- Judging summary qualitymedium
- The judge prefers answer Amedium
- A small prompt fix shipsmedium
- Which metric for auto-refundsmedium
- Vague 1-10 judge scoresmedium
- Perfect on 20 exampleshard
- Can we trust the LLM judgehard
- Switching to a newer modelhard
- Invented values in extractionhard
- Judge and candidate from one familyhard
- A suspicious score jumphard
- Is a 2-point gain real?hard
- Where is the RAG bug?hard
- Real sources, wrong claimshard
- Longer answers keep winninghard
- Tiny wording, big swinghard
- Errors flowing down a chainhard
- Scores change between runshard
- The model agrees with everythinghard
- Evals green, users unhappyhard
- Retesting vs regressioneasy