SimpleQA Verified leaderboard

What SimpleQA Verified measures

SimpleQA Verified is a factoid question-answering benchmark of 1,000 short questions on topics such as politics, science and technology, art, sports, geography and music. Each question has one answer that can be checked against a source. It is a cleaned-up version of OpenAI's SimpleQA, built to remove questions with wrong or ambiguous reference answers so that a wrong score reflects the model rather than the test.

The point of the test is narrow and useful: how much does a model reliably know without looking anything up, and how often does it state something false with confidence? Epoch AI reports the percentage answered correctly. A model that declines to answer when unsure is not rewarded as correct, but it also avoids the confident wrong answers that make hallucination costly in production.

How to read this page

The cost chart uses output price per 1M tokens as the cost axis. Unlike the math and agent benchmarks, a factoid answer is short, so the cost difference here comes mainly from price and any reasoning the model does before answering, not from long outputs. This benchmark rewards scale and training data more than reasoning effort, so a cheaper model that knows less is unlikely to close the gap by thinking longer.

If your application answers questions from your own documents or from a search tool, this score matters less than how well the model reads and cites the provided context. It matters most when the model must answer from memory.

Limitations

A 1,000-question set means gaps of a point or two are noise. The questions test obscure, discrete facts, so the score does not capture whether a model summarizes, reasons or writes accurately. The questions are public, which creates a contamination risk, and ambiguity or errors can remain even after verification. Results measure the model without web search or tools, so they understate what a search-enabled product would get right.