Big-Bench Hard leaderboard
Big-Bench Hard leaderboard: chain-of-thought accuracy across 23 hard reasoning tasks for each LLM, plotted against API price. Current leader: Gemini 1.5 Pro (May 2024) at 85.6 (% accuracy).
Cost vs quality
Each dot is a model. The dashed line is the value frontier — top-left is the best trade-off (high score, low cost).
Full leaderboard
OPEN
OPEN
OPEN
OPENOPEN
OPENOPENOPENOPENOPEN
OPENOPENOPENOPENOPENOPENOPENOPENScores from Epoch AI, updated —. Cost is joined from our daily API pricing where a model matches; models we don't price still appear with their score. Always verify on the benchmark's own site before relying on a number.
What Big-Bench Hard measures
Big-Bench Hard (BBH) is a 23-task subset of Google's original BIG-Bench suite, hand-picked in an October 2022 paper by Mirac Suzgun, Jason Wei, Denny Zhou and co-authors at Stanford University and Google Research specifically because no model of that era's few-shot performance beat the average human rater on these tasks. The tasks span logical deduction, multi-step arithmetic, algorithmic/symbolic manipulation (tracking shuffled objects, Dyck-language closure, word sorting), and causal or commonsense inference — each built to block shortcut heuristics and force genuinely compositional, multi-step reasoning. Models are run with the paper's original 3-shot chain-of-thought exemplars, and the final answer each model gives is checked by exact-match string comparison against the gold label.
Each of the 23 tasks contributes up to 250 held-out examples (a couple of tasks have fewer), for 6,511 problems total, and the reported BBH score is the plain average of the 23 per-task accuracies. The paper's headline finding was that chain-of-thought prompting — not just model scale — closed most of the gap between contemporary models and the human-rater baseline.
How to read this page
The cost axis is each model's output price per 1M tokens, used as a proxy because BBH has no published per-task dollar cost. That proxy fits BBH well because its protocol requires 3-shot chain-of-thought generation on every one of the 6,511 problems — the model must write out an explicit multi-step reasoning trace before its final answer, so completion tokens (plus any extended "thinking" tokens a modern reasoning model adds) dominate the token bill far more than the short few-shot prompt does. Read the leaderboard as ranked by score first, then use the chart to spot models delivering comparable BBH accuracy at a materially lower output-token price.
Limitations
BBH is a textbook case of benchmark saturation: it was built in 2022 to be hard for that era's models, but frontier models now cluster in the low-to-mid 90s% accuracy, so gaps between top models often sit inside the noise of a benchmark that caps most tasks at only 250 examples. Its exact-match scoring is also brittle against free-form chain-of-thought output: a correct reasoning trace whose final answer is formatted slightly differently than expected can be marked wrong, so scores are sensitive to the answer-extraction scaffold a given evaluation harness uses. Because the dataset and its canonical 3-shot CoT exemplars have been public on GitHub since 2022 and are baked into widely-used harnesses, training-data contamination is a standing, hard-to-rule-out concern for newer models. Finally, the 23 tasks are synthetic, template-generated puzzles selected for one narrow property (models of 2022 used to fail them); they say nothing about open-ended generation, tool use, or real-world reasoning.