A Dormant Hugging Face Hallucination Leaderboard Just Saw Downloads Double — and It Isn't Vectara's

On October 13, 2026, machine.net's internal download tracker flagged hallucinations-leaderboard/results as the fastest-growing dataset on Hugging Face by downloads — up from 381,155 to 794,256 in a single day, a 108% jump (source: ai-tracker, reports/2026-10-13.md, Hugging Face — Datasets, "Fastest-growing by downloads"). A spike like that on an obscure dataset repo usually means something happened: a model posted a bad hallucination score, a leaderboard got refreshed, a paper made news. We went looking for that trigger. We didn't find one — but we found something more useful: the project behind the dataset isn't who it's commonly assumed to be, and it looks largely abandoned.

It's not Vectara's leaderboard

The name invites the mix-up. Vectara, the retrieval-augmented-generation infrastructure company, runs a well-known "Hallucination Leaderboard" on GitHub and Hugging Face that ranks roughly 105 models on how often they invent facts while summarizing documents, scored by Vectara's own HHEM-2.3 detection model against a private 7,700-article dataset, and refreshed as recently as May 2026. That is a different, narrower project. hallucinations-leaderboard/results instead belongs to an academic effort described in a 2024 paper, "The Hallucinations Leaderboard: An Open Effort to Measure Hallucinations in Large Language Models," credited to researchers at the University of Edinburgh, University College London, Peking University, Together AI, and Hugging Face — including Hugging Face's Clémentine Fourrier, who also co-built its Open LLM Leaderboard. Vectara is not listed among the maintainers. Rather than one "hallucination rate," this project reports two separate scores — factuality (does an unprompted answer match reality) and faithfulness (does the model stay true to a supplied source) — each averaged across 15 existing benchmarks (including Natural Questions, TriviaQA, TruthfulQA, FEVER, XSum, RACE, SQuADv2, and HaluEval) run through EleutherAI's LM Evaluation Harness. The authors evaluated open-source models only, citing cost, and explicitly flagged their own prompt templates and in-context example counts as under-explored — an important caveat against reading the rankings as definitive.

A spike with no obvious trigger

This is where the story gets more interesting than a simple viral-leaderboard angle. The dataset's public commit history shows no activity since late October 2024. Its companion Hugging Face Space — the actual visual leaderboard people would check for rankings — returned a server runtime error when we loaded it this week, and its discussion board has no threads from 2026 at all; the most recent substantive one is over a year old. Nothing we could find changed on the leaderboard itself around the date of the spike: no new model added, no refreshed scores, no accompanying post. Hugging Face's own current page for the dataset shows a monthly download count consistent with the tracker's post-spike figure, which corroborates that the number itself is real. But a real number without a real cause on the leaderboard side is still worth treating skeptically — a sudden doubling of downloads to a stale, erroring repository looks more like the signature of automated or bulk consumption (a dataset-aggregation pipeline, a mirror, a crawl feeding someone else's training or eval set) than a wave of researchers discovering fresh results.

Why hallucination leaderboards are under scrutiny anyway

Set the spike aside, and 2026 has still been a rough year for leaderboard trust generally, with hallucination measurement near the center of it. A widely discussed OpenAI and Georgia Tech paper, "Why Language Models Hallucinate" (arXiv:2509.04664), argues the problem is structural rather than mysterious: most benchmarks grade answers as simply right or wrong, so a model that guesses confidently scores better than one that admits uncertainty — and that scoring incentive, baked into the benchmarks leaderboards are built on, is what keeps producing hallucinations across the industry. Separately, "The Leaderboard Illusion" (arXiv:2504.20879) documented how a major public arena-style leaderboard let some providers privately test dozens of model variants and disclose only the best-scoring ones, skewing rankings in ways unrelated to real capability. Neither paper is about this specific dataset, but both describe the kind of structural weakness that makes any single hallucination leaderboard — this one, Vectara's, or whatever comes next — a signal worth weighing against other evidence rather than a verdict worth citing alone. Even Vectara's own team hedges this way, noting that its summarization-only metric "does not capture every form of hallucination" and should be read alongside other evaluations. hallucinations-leaderboard/results remains one of the few open attempts to score factuality and faithfulness across open models on a shared set of benchmarks, and its raw results still get pulled into other people's research — which may be the most mundane, and most likely, explanation for why its download count just doubled.