hallucination-leaderboard
Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documents
What is hallucination-leaderboard?
Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documents
Overview of hallucination-leaderboard
What it is
The Hallucination Leaderboard is a public leaderboard that compares LLM performance at producing hallucinations when summarizing short documents. It is computed using Vectara's Hallucination Evaluation Model (HHEM-2.3). The leaderboard was last updated on October 22, 2026.
Key features
- Uses HHEM-2.3, Vectara's commercial hallucination evaluation model, to compute rankings.
- An open-source variant, HHEM-2.1-Open, is available on Hugging Face and Kaggle.
- The dataset contains over 7700 articles from sources including news, technology, science, medicine, legal, sports, business, and education.
- Articles range from 50 to 24,000 words, covering both low and high complexity.
- The dataset is not publicly available to avoid overfitting by any LLM.
- Metrics reported include Hallucination Rate, Factual Consistency Rate, Answer Rate, and Average Summary Length.
Use cases
- Comparing the hallucination tendency of different LLMs when summarizing documents.
- Evaluating factual consistency in summarization tasks.
- Supporting research on hallucination and factual consistency in language models.
Getting started
See the repository README for details on accessing the leaderboard and using the HHEM model.
License and pricing
- License: Apache-2.0
- Pricing: open source
Based on the project's public README and repository metadata.