OpenAI says an audit of SWE-Bench Pro found widespread task-quality problems, estimating that about 30% of tasks are broken. The company has withdrawn its earlier recommendation to adopt the coding benchmark and says flawed evaluations can distort comparisons between AI coding systems.
OpenAI said it has withdrawn its earlier recommendation to adopt SWE-Bench Pro after auditing the coding benchmark and finding widespread task-quality problems.
In a post titled “Separating signal from noise in coding evaluations,” OpenAI said its analysis of SWE-Bench Pro found issues serious enough to affect the reliability of results reported on the benchmark. According to OpenAI, the audit estimated that roughly 30% of SWE-Bench Pro tasks are broken.
SWE-Bench Pro is part of a growing category of coding evaluations designed to test whether AI models can complete software-engineering tasks. Such benchmarks are often used to compare model performance on realistic programming work, including debugging and modifying codebases. OpenAI’s post argues that when tasks are incorrectly specified, poorly designed, or otherwise flawed, scores can become misleading rather than informative.
OpenAI said it no longer recommends adopting SWE-Bench Pro in its current form. The company framed the decision as a correction to its prior position, saying the benchmark’s task problems could distort how AI coding systems are evaluated.
Coding benchmarks have become an important way for AI developers, enterprise buyers, and researchers to compare systems that write or modify software. But OpenAI’s analysis underscores a broader problem: a benchmark’s usefulness depends not only on how difficult its tasks are, but also on whether those tasks have clear, valid, and verifiable answers.
OpenAI said broken tasks can lead to inaccurate conclusions about model capability. If an evaluation includes many flawed problems, a model may be penalized for failing tasks that are not actually solvable as written, or rewarded for behavior that does not correspond to correct software engineering work. That makes it harder to distinguish genuine progress from artifacts of the test set.
The issue is especially important for coding models because small differences in task setup can change results. A benchmark problem may depend on a specific repository state, test behavior, dependency setup, or interpretation of a bug report. If those assumptions are wrong, incomplete, or inconsistent, benchmark scores can lose their practical meaning.
Investing.com reported that OpenAI had retracted its SWE-Bench Pro recommendation after an internal audit found that about 30% of tasks contained design flaws capable of distorting AI coding results. Techmeme also summarized OpenAI’s post as saying the company found widespread task issues in SWE-Bench Pro, estimated that around 30% of tasks were broken, and withdrew its earlier recommendation.
Those summaries align with OpenAI’s own statement that the benchmark should not be relied on in its current form. The company’s post does not mean coding evaluations are unusable, but it does point to the need for more careful validation before benchmarks become widely adopted as measures of model quality.
OpenAI’s decision adds to ongoing scrutiny of AI evaluation methods. As model developers compete on public scores, benchmark design has become a central part of how the industry communicates progress. A flawed benchmark can influence product decisions, research priorities, and customer expectations.
The practical takeaway from OpenAI’s post is that benchmark results should be read with attention to task quality, not just leaderboard position. For coding systems in particular, evaluations need well-specified tasks, reliable test cases, reproducible environments, and review processes that catch invalid examples before scores are used to compare models.
OpenAI’s retraction of its SWE-Bench Pro recommendation is therefore less a rejection of coding benchmarks as a category than a warning about how easily they can mislead when task design breaks down.
OpenAI said it has withdrawn its earlier recommendation to adopt SWE Bench Pro after auditing the coding benchmark and finding widespread task quality problems.
According to OpenAI, the audit estimated that roughly 30% of SWE Bench Pro tasks are broken.
SWE Bench Pro is part of a growing category of coding evaluations designed to test whether AI models can complete software engineering tasks.
Continue reading
