hard20 pts · first answer countstransformerscontext-windowevaluation
Passing the needle test is not enough
A model scores 100% on "needle in a haystack" tests at 200K tokens. Your task needs it to combine facts spread across 50 contracts in one prompt, and results are poor. What does this show?
Discussion 0
Join free to post in the discussion.
Related questions
- Why keep a separate test setAI Engineer (LLM & Generative AI)
- Computing precisionAI Engineer (LLM & Generative AI)
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- What a golden set is forAI Engineer (LLM & Generative AI)
- Does the reply use context spaceAI Engineer (LLM & Generative AI)
- Checking prompt size in advanceAI Engineer (LLM & Generative AI)
- Computing precision at 5AI Engineer (LLM & Generative AI)
- What retrieval hit rate measuresAI Engineer (LLM & Generative AI)