Prompt Engineer
LLM evaluation questions
25 questions on LLM evaluation for prompt engineers, easiest first.
Easy 5
Medium 5
Hard 15
- Perfect on 20 exampleshard
- Can we trust the LLM judgehard
- Switching to a newer modelhard
- Invented values in extractionhard
- Judge and candidate from one familyhard
- A suspicious score jumphard
- Is a 2-point gain real?hard
- Where is the RAG bug?hard
- Real sources, wrong claimshard
- Longer answers keep winninghard
- Tiny wording, big swinghard
- Errors flowing down a chainhard
- Scores change between runshard
- The model agrees with everythinghard
- Evals green, users unhappyhard