Hard Prompt Engineer questions
32 questions · 20 points each for a correct first answer.
- 1.Instructions hidden in user contentprompt-injection, safetyhard
- 2.Perfect on 20 examplesevaluationhard
- 3.Can we trust the LLM judgellm-as-judge, evaluationhard
- 4.Switching to a newer modelevaluation, model-migrationhard
- 5.Prompting a reasoning modelreasoning-promptshard
- 6.Adding 30 more examplescost-optimisation, few-shot-promptinghard
- 7.Invented values in extractionstructured-output, hallucination-reductionhard
- 8.Agent picks the wrong tooltool-use, instruction-designhard
- 9.Injection through a web pageprompt-injection, agents, llm-safetyhard
- 10.Leaking data through an image linkprompt-injection, llm-safetyhard
- 11.Prompt cache hit rate near zeroprompt-caching, prompt-debugginghard
- 12.Judge and candidate from one familyllm-as-judge, evaluationhard
- 13.A suspicious score jumpevaluation, golden-setshard
- 14.Is a 2-point gain real?evaluation, metricshard
- 15.Follow-up questions fail in RAGrag, query-rewritinghard
- 16.Where is the RAG bug?rag, evaluation, prompt-debugginghard
- 17.Strict JSON hurt accuracystructured-output, chain-of-thoughthard
- 18.Forced into the wrong labelstructured-output, classificationhard
- 19.An agent that deletes dataagents, guardrailshard
- 20.Hundreds of fake dialoguesjailbreaks, llm-safetyhard
- 21.Agent with 120 toolstool-calling, agentshard
- 22.Real sources, wrong claimscitations, hallucination, evaluationhard
- 23.Longer answers keep winningllm-as-judge, evaluationhard
- 24.Tiny wording, big swingevaluation, prompt-robustnesshard
- 25.Errors flowing down a chainprompt-chaining, evaluationhard
- 26.Scores change between runsevaluation, metricshard
- 27.Reasoning model bill surprisemodel-selection, cost-optimization, reasoninghard
- 28.The model agrees with everythingsycophancy, prompt-debugginghard
- 29.API key in the system promptllm-safety, prompt-injectionhard
- 30.Fresh facts plus a fixed stylerag, fine-tuning, model-selectionhard
- 31.Long agent run fills contextcontext-window, agentshard
- 32.Evals green, users unhappyevaluation, observabilityhard