AI coding benchmarks explained: SWE-bench, Terminal-Bench, LiveCodeBench, arenas, and what they can't show

What is an AI coding benchmark?

An AI coding benchmark is a fixed set of programming tasks with an automatic way to check the answers, run against many models so their scores can be compared. The check is the important part. Most benchmarks grade by running tests: if the model's code makes the hidden tests pass, the task counts as solved. That makes scores cheap to produce and easy to compare, and it also defines what they can see. Anything a test cannot check, such as readability, design or how a product feels to use, is invisible to the score.

What do the main coding benchmarks measure?

Each one measures a different slice of programming work.

SWE-bench Verified is a human-filtered subset of 500 instances, according to its maintainers. LiveCodeBench keeps collecting new problems from LeetCode, AtCoder and Codeforces precisely so that models cannot have seen the answers during training.

How should you read a coding benchmark score?

Ask four questions before believing any number.

1. Which harness? The same model scores differently inside different agents. The public Terminal-Bench leaderboard lists every entry with its agent, such as Claude Code or Codex, for exactly this reason.
2. Which effort setting? Reasoning models can be run at low to maximum effort. A score at maximum effort can cost several times more per task than one at medium.
3. Who ran it? A vendor's launch chart is run by the vendor, on settings it chose, against competitors it picked.
4. What did it cost? Two models with the same score can differ fivefold in price per task. Good leaderboards publish cost next to the score.

Why do top models score so high on older benchmarks?

Because benchmarks saturate. Once the strongest models solve nearly every task, a test stops separating them. OpenAI's own launch post for GPT-6 Astra reports 98% on FrontierMath Tier 4 and 99.9% on ARC-AGI-3, both of which were considered far out of reach a year earlier. On the public SWE-bench Verified bash-only leaderboard, the top entries cluster in the mid-70s, a few points apart, on runs dated early 2026.

Saturation has a quieter cousin, contamination: if a benchmark's answers circulate online long enough, they can end up in training data, and a high score then measures memory rather than skill. That is why LiveCodeBench keeps adding newly published contest problems and why newer suites such as Terminal-Bench ship fresh versions. When you see a near-perfect score, check the benchmark's age before you check the model.

Where do vendor numbers and public leaderboards disagree?

Often, and in instructive ways. OpenAI's GPT-6 Astra launch post reports Astra at 57.9% on Terminal-Bench 4.0, ahead of Claude Fable 5.1 at 55.8%. The public Terminal-Bench leaderboard, checked on October 22, 2026, lists Astra at 58.2% and Fable 5.1 at 57.9%, both with error margins of about three points, and puts Claude Opus 5.5 first at 64.8%. Opus 5.5 is absent from OpenAI's chart because it was released later. Neither source is wrong; they answer different questions on different days.

The same leaderboard also shows what the score hides. GPT-6.1 Sol ties Astra at 58.2%, but its run cost about $600 against Astra's $3,300. Opus 5.5 used about 8 billion tokens to reach its score, against 1.5 billion for each GPT model.

What can't coding benchmarks show?

The things you will actually notice. No test suite checks whether a landing page looks like a brand or a template, whether a dashboard's dark mode was designed or merely inverted, or whether a shooter feels good to play. Those qualities decide whether you keep a build, and they are exactly what our briefs are built to expose. On our landing page brief, the two GPT models produced near-identical pages, down to the same two fonts, something no benchmark would register.

Benchmarks also miss behaviour over time: how long an agent works, how often it checks itself, when it stops. Our run logs show the Claude agents running 9 to 118 shell commands per build and the GPT agents 10 to 33, a difference that shapes cost and results but never appears in a pass rate.

What do benchmarks say about cost?

Increasingly, a lot, and it is often the most useful column. A pass rate tells you what a model can do; the cost column tells you what it will cost you to find out. On the public Terminal-Bench 4.0 leaderboard, the top Claude and GPT entries sit within about seven points of each other while their run costs range from about $600 to $6,200.

Our own builds show the same spread at the scale of a single project. At list API prices, one build cost between $0.54 and $24.76 depending on the model, with the Claude agents at the top of that range because they re-read and re-test their work many times. A benchmark that reports only the score would rank the $20 build and the $1 build as if price did not exist. When you compare models for your team, multiply the score difference by how many tasks you run.

How do you test coding AIs beyond benchmarks?

Give them your kind of task, under the same conditions, and look at the results side by side. That means the same prompt word for word, one attempt each, a time limit, and no edits before you compare. It is the method our methodology page documents, and it is what the blind test below lets you try in a minute. Benchmarks narrow the field to a few capable models. Your own side-by-side picks the winner for your work.

Which benchmark should you trust for your use case?

Match the benchmark to the job. For maintaining an existing codebase, weight SWE-bench Verified. For autonomous agents, look at Terminal-Bench with its harness column. For interviews and algorithms, LiveCodeBench. For anything visual or interactive, prefer human judgement: arena votes in aggregate, or your own eyes on real builds. And for every number, find the date: the leaderboards above moved several times in the six weeks after GPT-6 and Claude 5.5 shipped.

Frequently asked questions

What is the best AI coding benchmark?

There is no single best one. SWE-bench Verified is the most cited for fixing real bugs, Terminal-Bench for agentic work in a terminal, LiveCodeBench for fresh contest problems, and arena-style sites for human preference on generated apps. Use the one closest to your own work.

Why do vendor benchmark numbers differ from public leaderboards?

Vendors run their own harnesses, settings and subsets, and choose which competitors to show. On Terminal-Bench 4.0, for example, OpenAI's launch post reports GPT-6 Astra at 57.9%, while the public leaderboard lists it at 58.2%, with Claude Opus 5.5 ahead at 64.8%.

Can a benchmark tell me which AI writes the best website or game?

Not directly. Most coding benchmarks check whether tests pass, not whether a page looks designed or a game is fun. Arena sites capture human taste through votes, and side-by-side builds let you judge it yourself.

Are coding benchmark scores comparable across models?

Only when the harness, effort setting and task set are the same. A model run in its vendor's own agent at maximum effort can score very differently from the same model in a generic harness at default settings.