SWE-bench

SWE-bench and Terminal-Bench: From Yardstick to Defendant

Every popular coding benchmark seems to follow the same arc. It launches hard, gets saturated, gets audited, and gets replaced. The interesting part is what each audit found, and what that means for anyone choosing a coding agent from a leaderboard.

Updated October 7, 2026 · Sources are linked at the bottom · Leaderboard figures are cited with their version and harness wherever possible

59.4%of 138 hard SWE-bench Verified tasks OpenAI audited had flawed tests or descriptions

~30%of SWE-bench Pro’s 731 public tasks judged broken in OpenAI’s July 2026 audit

28 of 89Terminal-Bench 2.0 tasks (31%) found defective and fixed in 2.1

29.9 ppmaximum accuracy swing from changing only the harness, same OpenAI model

The short version

SWE-bench asks a model to fix a real GitHub issue. Terminal-Bench asks an agent to finish a real job inside a command line. Different tasks, same problem: the score is only as trustworthy as the tests behind it, the data the model has seen, and the harness wrapped around it.

The finding that matters: in both cases the benchmark’s own maintainers or its heaviest users found the defects, published them, and shipped fixes. That is the healthy version of the story. The unhealthy version is the leaderboard number that keeps circulating after the audit.

Timeline

  • Oct 2023Original SWE-bench: resolved GitHub issues from 12 open-source Python repositories, each paired with its pull request and tests.
  • Aug 2024SWE-bench Verified: 500 problems kept from 1,699 reviewed, each checked by three engineers.
  • Nov 7, 2025Terminal-Bench 2.0 launches with 89 tasks, designed to keep frontier models under 50%.
  • Feb 23, 2026OpenAI stops reporting SWE-bench Verified and recommends SWE-bench Pro.
  • Jul 8, 2026OpenAI audits Pro, estimates roughly 30% of tasks broken, and withdraws that recommendation.
  • 2026Terminal-Bench 2.1 fixes 28 tasks; 3.0 arrives as a rolling task set; by Aug 28, 4.0 removes 8 tasks and revises 20.

Case A: SWE-bench, the benchmark that outgrew its own tests

What it measures

The model sees an issue and a repository snapshot. It does not see the tests. It passes only if the hidden fail-to-pass tests succeed and the regression tests still hold. That design is elegant, and it is also the weak point: the tests are a stand-in for “correct”, and a stand-in can be wrong.

What the audit found

OpenAI’s February 2026 analysis noted that state-of-the-art progress on Verified had slowed from 74.9% to 80.9% over six months, then asked whether the remaining failures were model limits or dataset limits. It audited 138 problems that o3 failed to solve consistently across 64 runs, with at least six engineers reviewing each one. The result: 59.4% had material problems. Of those, 35.5% had narrow tests that enforce an implementation detail the issue never mentioned, and 18.8% had wide tests that check functionality the issue never asked for.

The second finding is worse for anyone reading a leaderboard. When prompted with little more than a task ID or a snippet, every frontier model OpenAI probed, including its own GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash, could reproduce gold-patch details or verbatim problem text for some tasks. The post’s conclusion is blunt: gains on Verified increasingly reflect exposure to the benchmark rather than software skill.

The sequel problem

SWE-bench Pro, built by Scale AI, was the proposed fix. Epoch AI describes it as 1,865 tasks: 731 public, 276 private, 858 held out. It cut contamination, but on July 8, 2026 OpenAI published a datapoint-level audit of the public split. Its automated pipeline flagged 200 of 731 tasks (27.4%) and human reviewers flagged 249 (34.1%), giving the roughly 30% estimate. Epoch now labels the benchmark “Flawed”. The failure types were familiar: misleading prompts, overly strict tests, underspecified prompts and low-coverage tests.

Case B: Terminal-Bench, a benchmark that treats itself like software

What it measures

Terminal-Bench is a Stanford and Laude Institute project. Each task ships with its own container, a human-written solution and verification tests, and agents work interactively through a terminal. For 2.0, the team started from 229 candidate tasks and kept 89 after a difficulty assessment and review by three experienced humans. Tasks run through the Harbor harness, and every leaderboard entry carries a 95% confidence interval.

What the audit found

A paper titled “Benchmarks as Software: A Case Study on Terminal-Bench” audited 2.0 with continuous validation: automated CI checks plus LLM-as-judge review. It found task-side defects in 28 of 89 tasks (31%). Nine came from environment drift, because tasks with internet access depended on things that changed after release. Eight had instructions that did not match the tests. Those fixes shipped as Terminal-Bench 2.1.

The numbers are what make this case study useful. Scores rose by up to 12 percentage points, but agent ranking barely moved (Spearman ρ = 0.958). And 28.8% of the tokens spent on failed trials on the changed tasks went into work that 2.1 confirms is solvable. In other words, the defects mostly shifted absolute scores and wasted compute rather than reshuffling the order.

The harness is half the score

On 2.1, the same OpenAI model scored up to 29.9 points lower when run under the terminus-2 scaffold instead of codex. Giving the agent more time per task closed the gap, which suggests the harness penalty was about wall-clock limits rather than capability. If you read one line of this article twice, make it this one: a score without a named harness is not a comparable number.

Saturation, then succession

The Terminal-Bench team says top agents on 2.1 already clear roughly 75–84%. Version 3.0 (formerly branded Frontier-Bench) is a community-built rolling set of 74 tasks across 7 domains, each passing rubric review, maintainer review and adversarial trials that probe for reward hacking. The project’s own page puts the best models near 34% at launch, and Snorkel AI reported a top result of 43.5%. Version 4.0, shipped by August 28, removed 8 tasks, revised 20 and added none. Maintenance became the product.

Side by side

SWE-bench (Verified / Pro)Terminal-Bench (2.0 → 4.0)
Unit of workFix one repo’s issue; hidden unit tests decideFinish an interactive terminal job in a container
Main defect foundTests that reject valid fixes; training exposureEnvironment drift; instruction/test mismatch
Share of tasks affected59.4% of hard subset (Verified); ~30% (Pro public)31% (28 of 89)
Who found itOpenAI, an outside lab and heavy userBenchmark’s own team and academic collaborators
ResponseAbandon, recommend successor, then withdraw that tooVersion, patch, publish deltas, add adversarial review

Five lessons you can use this week

  1. Treat every score as a triple: number, benchmark version, harness. If one is missing, discard the comparison.
  2. Prefer rankings over absolutes. Terminal-Bench’s own data shows fixes move scores far more than they move order.
  3. Distrust public task sets. If the problems and solutions are on the open web, assume some exposure, and weigh private or held-out splits more heavily.
  4. Look for a changelog. A benchmark that publishes removed and revised tasks, as Terminal-Bench 4.0 does, is behaving like maintained software. One that does not is a snapshot.
  5. Run your own evaluation. Take twenty real tickets from your codebase and your own verification checks. It costs a day and beats any leaderboard for your use case.

A caution on leaderboard numbers

While researching this piece, aggregators disagreed on the top Terminal-Bench 2.0 score. One tracker listed a GPT-5.5 result at 82–82.7%, another listed 84.3% for Claude Fable 5 and marked other entries unverified, and a third reported different leaders on a different task set. The disagreement is not a scandal. It is the harness and version problem showing up live. Check the official Terminal-Bench leaderboard and the model vendor’s own methodology note before quoting a number.

Sources