What's the difference between SWE-bench Verified and the original SWE-bench? Why did even the "fixed" version end up failing?
The original SWE-bench, released in 2023, had its own problems — unstable test environments, some problems that were essentially unsolvable. OpenAI brought in experts in 2024 to manually review 1,699 of those problems and filter them down to a higher-quality 500-problem subset called SWE-bench Verified, specifically to fix these obvious evaluation flaws.
But the reason this version failed is different from the original's problems — it's not primarily that the problems themselves were badly designed (though this audit did find residual test-design issues affecting nearly 60% of the audited set). It's contamination: all 500 problems come from public open-source projects, and as time passes, those codebases become increasingly likely to have been swept into the web-crawled training corpora various companies use. In other words, the "fixed" version solved the problem-design layer, but couldn't solve the structural problem that any public problem set will eventually be seen by the models being tested on it — which is exactly why even a carefully reviewed version couldn't escape the same fate.
If it's just "some problems have been seen by models," why was that serious enough for OpenAI to abandon the entire leaderboard?
The key issue isn't just whether a model has seen a problem — it's a point OpenAI's analysis specifically emphasizes: models that have seen the answers succeed at meaningfully higher rates, because they've gained extra information the problem description itself doesn't fully spell out — effectively getting the complete answer to an incompletely specified question. For example, the task django__django-14725 requires a new parameter called edit_only in its tests, but that parameter name never appears anywhere in the problem description. Yet GPT-5.2's reasoning trace stated outright that this parameter was "probably added around Django 4.1" — meaning it was actually answering from memorized knowledge of Django's version history, not genuinely reasoning its way to a solution from the problem description alone.
That means part of the apparent score improvement over time is really an improvement in memorization, not an improvement in the ability to solve genuinely unseen problems — and it's the latter that the industry actually wants to measure, and that the Preparedness Framework is actually trying to track. Once these two things are blended into a single number with no way to separate them, that number loses the purpose it was built for.
Does SWE-bench Pro actually solve the problem for good, or just delay the same dilemma?
In the short term, SWE-bench Pro does meaningfully reduce contamination — using the same extraction technique, OpenAI found far fewer successful contamination cases than on SWE-bench Verified, and no model could reproduce a complete gold patch. That's clear evidence its current protection is significantly stronger.
But structurally, SWE-bench Pro faces a later stage of the same long-term dilemma: being a private problem set solves the issue of public data being swept into training corpora, but at the cost of much higher build-and-maintenance overhead and a more limited pool of problem sources. Over time, if this problem set's contents leak into the public web in various ways — cited, discussed, even partially exposed — it could face the same contamination risk eventually. That's exactly why OpenAI is simultaneously emphasizing investment in evaluations like LiveCodeBench, which dynamically filters by publication date, and GDPVal, which uses fully private task authoring with human grading — layering multiple defense mechanisms together rather than pinning all hope on a single leaderboard.
Ordinary readers aren't researchers and can't independently verify contamination — so how should they judge whether a coding agent product's advertised score is trustworthy?
There are a few relatively verifiable clues worth watching for. First, check whether the score a company cites is SWE-bench Verified — if it is, and there's no accompanying note about contamination risk or an alternative metric, that's itself worth treating as a yellow flag, since the industry standard shifted after February 2026. Second, check whether the company also publishes a score on SWE-bench Pro or another contamination-resistant leaderboard, and compare the gap between the two — a wide gap means the Verified score has limited reference value. Third, if a product's marketing offers only a single number from a single leaderboard, with no methodology notes or links to primary sources, that presentation itself violates the basic principle that any single score needs cross-checking — worth a few extra questions before making a decision based on it.
On February 23, 2026, OpenAI published a rare self-correcting statement: the company would stop reporting SWE-bench Verified scores and recommended the rest of the industry do the same. What makes this worth paying attention to isn't that another benchmark quietly aged out — the AI world has grown fairly numb to leaderboards getting gamed — it's that SWE-bench Verified was itself the "fixed" version OpenAI built in 2024 to correct flaws in the original SWE-bench, and now even that fixed version has been declared unreliable by its own creator.
When SWE-bench first launched in 2023, it was a genuine step forward in AI coding evaluation — instead of testing toy-scale single-function completions, it dropped models directly into real GitHub issues from real open-source projects, requiring them to read a problem description, modify the corresponding code, and pass the tests that shipped with the original pull request. Early top scores hovered around 20-30%, and the benchmark was for a while regarded as hard, honest, and genuinely useful. But the original version had clear evaluation flaws — overly rigid test cases, ambiguous problem descriptions, unstable cross-platform execution environments — so in 2024 OpenAI brought in experts to manually review 1,699 original problems and filter them down to a curated 500-problem subset, released as SWE-bench Verified.
Over the following eighteen months, SWE-bench Verified quickly became an industry-standard metric, appearing in nearly every frontier model release and used internally by OpenAI's own Preparedness Framework to track capability progress. But according to OpenAI's newly published analysis, top-model improvement on this benchmark has slowed from 74.9% to 80.9% over the past six months — which pushed the company to ask a more fundamental question: do the remaining unsolved problems reflect a genuine capability ceiling, or a flaw in the dataset itself?
OpenAI's audit focused on 138 problems models frequently failed to solve, with each one independently reviewed by at least six experienced engineers. The results showed 59.4% of these problems had material defects: 35.5% suffered from "narrow test cases" — tests demanding specific implementation details (like requiring a function to be named a particular way), so a functionally correct alternative solution still fails on a naming mismatch. Another 18.8% suffered from "wide test cases" — tests covering more ground than the problem description asked for, such as a case where the original pull request actually fixed three separate issues but the problem description only extracted one of them; a model correctly solving the described issue would still fail because it didn't also fix the two "unrequested" issues bundled into the same tests.
The second, more fundamental problem is contamination: every SWE-bench problem is sourced from public open-source projects, and those codebases, release notes, and even discussion threads have likely already been swept into various models' training corpora. OpenAI built an automated red-teaming setup where GPT-5 tried to "extract" answers from non-reasoning models — GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash Preview — by giving them only a partial hint about a problem and seeing whether they could "recall" the complete correct fix (the gold patch). The results were striking: given just a short fragment of a problem description, GPT-5.2 reproduced the exact code change in full, down to the precise function and variable names; Claude Opus 4.5 could quote the original pull request's inline code comment verbatim; Gemini 3 Flash was able to recite the problem description word-for-word. Three different companies, three different models, all caught having seen the correct answers to the same set of problems.
OpenAI has now publicly recommended the industry switch to SWE-bench Pro as a replacement metric — a privately held problem set specifically designed to resist contamination. OpenAI's own contamination testing found the same extraction technique surfaced far fewer contamination cases on SWE-bench Pro than on SWE-bench Verified, and no model was able to reproduce a complete gold patch verbatim. But switching to the new leaderboard comes with scores that appear to fall off a cliff: models scoring above 80%, even approaching 93%, on SWE-bench Verified typically land between 46% and 58% on SWE-bench Pro. That gap is itself the clearest piece of evidence — models didn't suddenly get worse; the old leaderboard had been systematically overstating progress all along.
The bind SWE-bench has fallen into is a dilemma every benchmark built from public data eventually faces: the closer a problem source is to the real world, the more likely it is to end up in training data; the more closed and private a problem source is, the more expensive it becomes to build and maintain. OpenAI's analysis acknowledges this is exactly why the company has recently shifted toward evaluations like GDPVal, where tasks are privately authored by domain experts and graded holistically by trained human reviewers — far more expensive, but effective at cutting contamination risk. This echoes an emerging industry consensus flagged by several independent analyses recently: before reading any benchmark score, you first have to know what "defense mechanism" that score actually has — does the problem set refresh over time, is it re-authored for specific domains, is scoring automated or human-graded — because a score with no defense mechanism is, fundamentally, just a set of numbers waiting to be gamed.
If you're evaluating an AI coding tool, or making an AI-related investment decision based on benchmark scores, the most direct takeaway from this episode is that no single leaderboard score should be treated as a stable, trustworthy thermometer reading — it's closer to an opinion poll that needs cross-checking. Concretely: check whether a company is still reporting SWE-bench Verified scores today without any caveat (that may signal less transparent disclosure practices); compare the same model's score gap between SWE-bench Verified and SWE-bench Pro — the wider the gap, the less the old score is worth as a reference point; and over the longer term, pay attention to evaluations like LiveCodeBench that dynamically refresh their problem sets by publication date specifically to prevent training-data contamination — these aren't perfect either, but they at least make "what's the defense mechanism" a verifiable question rather than something you just have to take a vendor's word for.