MMLU leaderboard
MMLU leaderboard: 57-subject multiple-choice knowledge accuracy for each LLM, plotted against API price so you can weigh broad knowledge against cost. Current leader: GPT-4o (Nov 2024) at 84.13 (% accuracy).
Cost vs quality
Each dot is a model. The dashed line is the value frontier — top-left is the best trade-off (high score, low cost).
Full leaderboard
OPEN
OPEN
OPEN
OPENOPENOPENOPENOPEN
OPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENOPENScores from Epoch AI, updated —. Cost is joined from our daily API pricing where a model matches; models we don't price still appear with their score. Always verify on the benchmark's own site before relying on a number.
What MMLU measures
MMLU ("Measuring Massive Multitask Language Understanding") is a multiple-choice knowledge benchmark introduced in 2021 by Dan Hendrycks and coauthors at UC Berkeley, comprising roughly 15,900 four-option questions (A–D) spread across 57 subjects — elementary math, US history, computer science, law, clinical medicine, virology, marketing, and more — pulled from real practice exams, textbooks, and quizzes rather than authored specifically for the benchmark. A model's % accuracy is simply the share of test questions where its selected letter matches the answer key, macro-averaged across subjects into a single overall score; random guessing yields roughly 25%.
Scoring can be done two ways depending on the evaluation harness: the paper's original method compares the log-probability a model assigns to each of the four candidate letters and takes the highest, while most current model-card evaluations instead let the model generate a free-form response (often after chain-of-thought reasoning) and extract the final letter programmatically. Because MMLU places no strict requirement on which method or shot-count is used, aggregators collect scores from a mix of developer self-reports and independent reproductions, which is one reason near-identical models can show slightly different published MMLU numbers.
How to read this page
Rank by MMLU accuracy first and treat the cost-vs-quality chart as a secondary, imperfect lens: MMLU's per-question footprint is dominated by the input (a question stem plus four answer choices), and the graded output is nominally just one letter, so strictly this is an input-heavy task where output price isn't the natural cost driver. The proxy still holds up reasonably well in practice, though, because most frontier labs now score MMLU using chain-of-thought prompting rather than the paper's original direct-answer format — meaning a meaningful share of each model's generated (and billed) tokens is reasoning text produced before the final letter, not just the answer itself.
Limitations
MMLU has three well-documented weak points. First, dataset quality: a widely-cited 2024 audit manually re-annotated 3,000 questions and found an overall ~9% error rate — mislabeled answer keys, ambiguous stems, multiple defensible correct options — with per-subject error rates ranging from near 0% in fields like formal logic to as high as 57% in virology. Second, contamination: MMLU's questions were scraped from freely available textbooks, quizzes, and practice exams that have sat on the open web since 2021, so they've likely been ingested into later models' pretraining data, inflating scores through memorization rather than genuine generalization. Third, saturation: frontier models have pushed past the ~89.8% average human-expert score cited in the original paper and now cluster in the 88–92% band, compressing the benchmark's usable range and making small score gaps unreliable signals of real capability differences — which is why harder successors like MMLU-Pro and GPQA Diamond exist alongside it.