Prompt Engineer
KPIs and business metrics questions
36 questions on KPIs and business metrics for prompt engineers, easiest first.
Easy 11
- What exact match measureseasy
- Accuracy from four countseasy
- What a false positive iseasy
- 99% accuracy that is uselesseasy
- Tracking broken JSONeasy
- What perplexity tells youeasy
- A business metric for a support boteasy
- Computing retrieval hit rateeasy
- Computing precision at 5easy
- Recall of a document findereasy
- Replies slowly getting longereasy
Medium 15
- Which metric for auto-refundsmedium
- Computing F1medium
- Partial credit with token F1medium
- Macro or micro F1medium
- Agreement beyond chancemedium
- High ROUGE, wrong summarymedium
- Low BLEU for a good translationmedium
- Cosine similarity as a gradermedium
- Win rate when there are tiesmedium
- Measuring tool callingmedium
- Extraction scored too harshlymedium
- Measuring hallucinations more finelymedium
- Finding which labels get mixed upmedium
- Flagging chats for human reviewmedium
- Choosing a confidence thresholdmedium
Hard 10
- Is a 2-point gain real?hard
- Scores change between runshard
- Computing pass@2hard
- Succeeding every time, not oncehard
- Is the model's confidence honesthard
- How well the judge finds failureshard
- Scoring extracted line itemshard
- Average precision for one queryhard
- Auto-approving refunds safelyhard
- Comparing models by error costhard