A coding agent benchmark you can watch: Claude Code vs Codex on 8 briefs

What does this coding agent benchmark measure?

The model and the harness together, on real builds. A coding agent is a model wrapped in tools: it can read and write files, run shell commands and, if it chooses, open its own work in a browser. That harness changes results as much as the model does, so we ran each vendor's models in the vendor's own agent, the way developers use them: Claude Opus 5.5 and Claude Fable 5.1 in Claude Code, GPT-6 Astra and GPT-6.1 Sol in Codex, all at maximum reasoning effort.

For every build we publish what can be measured without opinion:

• Completion. Did the agent finish, and was every required feature present?
• Wall time. Minutes from start to finish, with a 150-minute cap.
• Tokens and cost. Output tokens, and what the run would have cost at list API prices.
• Runtime errors. Console and page errors while our recorder used the build.
• Interruptions. Infrastructure failures such as rate limits, logged as prior attempts.

What are the eight briefs?

Eight jobs developers really hand to agents, each with fixed content so that differences come from execution: a marketing landing page, an analytics dashboard, a physics simulation, a 3D shooter, a phone finance app, a one-lane MOBA, an open brief where the agent invents a product, and a kanban board. Each build must be a single HTML file with no build step, with local script files allowed for the games, so it can be opened years from now.

What does each brief stress?

Each one pushes on a different part of an agent's skill set.

How did Claude Code and Codex perform?

As two different strategies. Codex ran every build as a single long agent turn of 10 to 33 shell commands, finished each in 7 to 25 minutes, and at list prices cost $0.54 to $1.12 per build with GPT-6.1 Sol and $2.12 to $7.27 with GPT-6 Astra. Claude Code ran builds as many short turns, writing, serving, looking and fixing, and took 25 minutes to the full 150 per build, at roughly $7 to $25 each where it recorded a total. Claude Opus 5.5 reached the 150-minute limit twice, on the shooter and the MOBA.

Here is one brief in full. Every brief page carries the same table.

What did the benchmark find that a pass rate would miss?

Three things no automatic grader would have flagged.

1. Sameness. Codex's two models repeatedly built near-identical projects: the same landing page typefaces, physics demos with the same name, shooters with the same door marked Relay. A pass rate counts both as passes.
2. Robustness under play. Our scripted input killed the player in games from both vendors, including Claude Opus 5.5's shooter and both GPT MOBAs. Every one of those games passed its functional checks; a pass rate would never show which game is forgiving.
3. Cost of thoroughness. Claude Code's builds were richer, and its self-testing showed: Claude Opus 5.5 issued 118 shell commands on the shooter alone. That thoroughness is invisible in a pass rate and very visible on an invoice.

What did the hardest brief show?

The one-lane MOBA, the largest build we ask for, separated the two strategies most clearly. Codex finished both of its MOBAs in 20 and 25 minutes, with three to five script files each. Claude Code spent 131 minutes on Fable 5.1's and the full 150 on Opus 5.5's, with fourteen and seventeen files, shops, minimaps and named enemy heroes. All four games run; how much more the Claude lanes are worth is exactly the judgement the recordings leave to you.

How does this compare with Terminal-Bench?

It points the same way. The public Terminal-Bench 4.0 leaderboard, which also runs each model in its own agent at maximum effort, lists Claude Opus 5.5 in Claude Code at 64.8%, GPT-6 Astra and GPT-6.1 Sol in Codex at 58.2%, and Claude Fable 5.1 at 57.9%. It also reports that Opus used about 8 billion tokens for its run against 1.5 billion for each GPT model, the same pattern of thoroughness against economy that our builds show. For a guide to what Terminal-Bench, SWE-bench and others measure, see AI coding benchmarks explained.

How was the benchmark kept fair?

By publishing every rule and following it even when it hurt. The prompt is identical, word for word, for every agent. Personal configuration files, plugins and external tools are switched off: Claude Code runs with --safe-mode, Codex with --ignore-user-config. Each run happens in an empty folder outside any repository. Only infrastructure failures earn a retry, and every interrupted attempt is listed on its build with the reason. Recordings use the same script for every build, and games run on a virtual clock so a slow machine cannot make one look choppy. The one Claude run that hit the 150-minute limit was published as it stood, flagged on its build page.

What are the limits of this benchmark?

Several, and they are worth stating plainly. Each model made one attempt per brief, so we cannot tell you how much its results vary from run to run; a second attempt might be better or worse. Eight briefs are a small sample, and all of them are front-end builds in a single file, so they say little about backend work, large existing codebases or long refactors, where benchmarks such as SWE-bench are more informative.

Our measurements are also only what we can measure fairly. Runtime errors are counted while our script uses each build, so a bug off the scripted path would not appear. Costs are what each run would have cost at list API prices; our runs actually used subscriptions. And our recording scripts exercise the features each brief required, not everything a model chose to add. None of these limits favours one agent over the other, because every build was treated identically, but together they are why we publish builds rather than a ranking.

How should you use these results?

Pick the agent whose strategy matches your work. If you iterate quickly and review everything, Codex gives you a working base in minutes. If you want to hand off a build and come back to something finished, Claude Code is worth the time and tokens. Then check our conclusion against your own taste with the blind test, because the benchmark measures the work and only you can value it.

Frequently asked questions

What is a coding agent benchmark?

A test of an AI model running inside an agent harness, such as Claude Code or Codex, that can read files, run commands and check its work. It measures the model and the harness together, which is how developers actually use them.

How is this benchmark different from Terminal-Bench or SWE-bench?

Those benchmarks grade many small tasks automatically and report a pass rate. Ours runs eight large, open-ended builds, publishes the measurements for each, and leaves the quality judgement to you by recording every build and letting you open it.

Which coding agent did best?

Neither, overall, because they optimise for different things. Codex finished every build in under 25 minutes at a fraction of the cost; Claude Code produced richer builds and needed 25 minutes to two and a half hours each.

Can I reproduce this benchmark?

Yes. The eight prompts, the exact command lines for both agents, the 150-minute limit and the recording scripts are published on the methodology and brief pages.