Humanity's Last Exam leaderboard

What Humanity's Last Exam measures

Humanity's Last Exam (HLE) is a closed-ended academic benchmark built by the nonprofit Center for AI Safety (CAIS) and Scale AI, spearheaded by CAIS director Dan Hendrycks, explicitly to replace saturated tests like MMLU where frontier models already scored above 90%. Its roughly 2,500 questions were crowdsourced from nearly 1,000 subject-matter experts across 500+ institutions in 50 countries (competing for a $500,000 prize pool) and span 100+ subjects from mathematics and physics to classics and law. A question was accepted only if it first stumped the frontier LLMs available at submission time and then survived two further rounds of human expert review. About 76% of questions require an exact-match short answer, roughly 24% are multiple-choice with five or more options, and about 14% are multimodal, pairing text with a diagram, chart, or image.

Scoring is pass@1 percent accuracy: each model gets a single attempt per question, and its free-form response is extracted and graded against the canonical solution by an automated LLM judge rather than by human raters or simple string overlap. HLE's paper also separately reports an RMS calibration error, obtained by asking each model to state a 0-100% confidence alongside its answer; every model tested at launch showed severe overconfidence, with RMS calibration errors ranging from roughly 73% (the best-calibrated model tested) to about 89% — though this leaderboard surfaces only the headline accuracy figure.

How to read this page

The cost axis here is each model's output price per 1M tokens, used as a stand-in because HLE has no published per-question dollar cost — a reasonable one for this benchmark specifically. The final answer a model must produce is short (an exact-match string or a multiple-choice letter), but the models that actually score well on HLE are almost all reasoning models that spend thousands of tokens on an internal chain of thought before committing to that short answer, and those reasoning tokens are billed as output. So output price tracks the real cost of attempting an HLE-style question much better than input price would, even though the visible answer itself is brief.

Limitations

HLE's selection process kept only questions that already fooled frontier models and paired that with rushed human vetting — reviewers reportedly had as little as five minutes to check each answer rationale — and that combination affected accuracy: Scale AI's own internal audit found roughly an 18% error rate in the answer key, and an independent review of the biology/chemistry questions found about 29% of reference answers were wrong or misleading against the published literature. The organizers now run a rolling revision process to prune and replace flagged questions, but any given leaderboard snapshot may still reflect a dataset version with known-bad ground truth on some items. Beyond answer-key quality, HLE carries the usual risks of a static, published exam set — contamination as questions and discussion threads leak online, heavy sensitivity to prompting and reasoning-effort settings, and a modest sample size (2,500 questions) that makes subject-level breakdowns noisy — and scores moved fast even so: GPT-4o scored just 2.7% and OpenAI's o1 scored 8.0% at HLE's January 2025 launch, but a reasoning model in that same launch paper already reached 13%, and top models were scoring in the 30s by early 2026 — the same saturation dynamic the benchmark was built to outlast.