Terminal-Bench leaderboard

What Terminal-Bench measures

Terminal-Bench is an agentic-coding benchmark built by the Laude Institute (the AI research nonprofit founded by Databricks/Perplexity co-founder Andy Konwinski) together with Stanford researchers led by Mike Merrill, first released in 2025 and now maintained at version 2.x. It tests whether a model, wired up as an autonomous agent with nothing but a Linux shell inside a Docker container, can complete real-world-style terminal tasks end to end with no human in the loop — provisioning and configuring software, fixing broken builds, running data-science or ML pipelines, administering servers, and security/reverse-engineering exercises. Terminal-Bench 2.0 narrowed 229 tasks crowdsourced from 93 contributors down to 89 hand-verified tasks, each shipping a natural-language instruction, a purpose-built container environment, a human-written "oracle" solution, and an automated test suite that inspects the final container state to decide pass or fail. Models are run through Harbor, the project's open-source harness, either via Terminus — a deliberately bare reference agent that amounts to little more than an LLM wired to a single terminal tool, with no extra scaffolding — or via other agent scaffolds such as Claude Code, Codex CLI, and OpenHands.

The metric shown, % resolved, is the share of the task set a model's agent completes successfully as judged by each task's automated verifier, generally averaged over multiple repeated runs per task to smooth out the run-to-run variance inherent in long, multi-step agent trajectories. Within a given Terminal-Bench version, only the best-scoring model+agent-harness pairing for a given model is carried onto the public leaderboard.

How to read this page

The cost axis here uses each model's output price per 1M tokens as a stand-in for real per-task dollar cost, because Terminal-Bench doesn't publish one — and for this benchmark that's a reasonable proxy: solving a task means the agent grinds through a long multi-turn trajectory of shell commands, error output, re-planning, and (for reasoning models) extended chain-of-thought before it converges on a working solution, so the tokens the model generates dominate the trajectory far more than the short natural-language task prompt does. That makes output price the side of cost that actually moves with task difficulty here.

Limitations

Terminal-Bench's task bank is fully open-source on GitHub, and its only anti-contamination measure is a canary string embedded in each file — a symbolic safeguard at best, since labs can still train on (or overfit to) publicly visible tasks and solutions. The dataset is also small: 80 tasks in version 1.0 and 89 in version 2.0, which is part of why the benchmark reports confidence intervals around each score. Scores are notably scaffold-sensitive — the published paper reports the same underlying model shifting by roughly 17 percentage points depending on whether it's wrapped in Terminus 2 or OpenHands, meaning the leaderboard partly measures tool-use scaffolding rather than pure model capability. The benchmark has also saturated faster than expected: the top model+harness pairing resolved about 63% of tasks shortly after Terminal-Bench 2.0 launched in November 2025, and had already climbed past 75% by the time Terminal-Bench 2.1 shipped roughly six months later — though 2.1's stated purpose was fixing 28 of the 89 tasks broken by internet-dependency drift or mismatched instructions, rather than deliberately raising the difficulty bar. Results can also drift run-to-run because some tasks call out to the internet and aren't fully deterministic. Finally, its scope is narrow by design — purely command-line, containerized tasks — so it says little about GUI-based work, longer-horizon real-world software engineering with ambiguous specs, or multimodal agent capability.