AI Engineer (LLM & Generative AI)
LLM evaluation questions
43 questions on LLM evaluation for ai engineer (llm & generative ai)s, easiest first.
Easy 13
- Why keep a separate test seteasy
- Computing precisioneasy
- What a golden set is foreasy
- Computing precision at 5easy
- What retrieval hit rate measureseasy
- What a golden eval set containseasy
- What faithfulness measureseasy
- What answer relevance measureseasy
- Using thumbs-up and thumbs-downeasy
- What LLM-as-a-judge meanseasy
- Reading a perplexity numbereasy
- Why LLMs state false factseasy
- What a knowledge cutoff meanseasy
Medium 16
- Pairwise LLM judge biasmedium
- Computing recall at 5medium
- Computing MRRmedium
- Judge prefers longer answersmedium
- Picking an embedding modelmedium
- What context recall measuresmedium
- What context precision rewardsmedium
- Eval questions generated from chunksmedium
- Judge prefers whichever answer is firstmedium
- Can you trust the LLM judge?medium
- Offline versus online RAG evaluationmedium
- Small change, broken answersmedium
- The bot says "I don't know" too oftenmedium
- Measuring a support bot in productionmedium
- Building an eval set from logsmedium
- Turning a loss into perplexitymedium
Hard 14
- Shipping a prompt change safelyhard
- A judge with 90% agreementhard
- Computing nDCG at 3hard
- Is a 4-point gain realhard
- Fine-tuned model scores too wellhard
- Choosing a vector index configurationhard
- Setting the balance in hybrid searchhard
- Perfect faithfulness, unhappy usershard
- Suspiciously high eval scoreshard
- A 4-point gain on 50 questionshard
- Better offline, flat onlinehard
- Faithful answer, wrong documenthard
- Token probabilities after RLHFhard
- Passing the needle test is not enoughhard