Prompt Engineer
Eval Design questions
26 questions on Eval Design for prompt engineers, easiest first.
Easy 10
- Code checks or an LLM judgeeasy
- Grading with or without a referenceeasy
- Offline and online evaluationeasy
- A test set unlike real trafficeasy
- Comparing scores across monthseasy
- Pass/fail per criterioneasy
- How big a first eval seteasy
- Same question, different wordseasy
- When the answer key is wrongeasy
- Evals compared with unit testseasy
Medium 9
- Generating varied synthetic testsmedium
- Testing the awkward inputsmedium
- Testing whole conversationsmedium
- Two sides of a summarymedium
- Scores fine, complaints risingmedium
- Building tests from real chatsmedium
- Two kinds of eval suitesmedium
- A great answer with one dangerous errormedium
- Splitting logs into dev and testmedium