The authors argue that existing benchmarks for testing hallucination detectors fall short—they lack long-context RAG examples and don't simulate realistic label noise. They build TRIVIA+, a new benchmark with the longest context in the literature, human-annotated answers, and four different flavors of noisy labels (both sample-dependent and sample-independent). Testing popular hallucination detectors on TRIVIA+ reveals that current methods have a long way to go, basic LLM-as-a-judge baselines are surprisingly competitive, and label noise hurts performance.
Main takeaways:
- Existing hallucination benchmarks don't test detectors on long RAG contexts or under realistic label noise, limiting their usefulness.
- TRIVIA+ is a new RAG-based benchmark with the longest contexts available and four curated noise schemes for stress-testing detectors.
- Current state-of-the-art detectors leave significant headroom for improvement on RAG tasks.
- Simple LLM-as-a-Judge baselines perform competitively with more complex methods.
- Label noise—whether from human annotators or automated labeling—degrades detector performance, highlighting robustness gaps.