OpenAI says an audit of SWE-Bench Pro found serious task-quality problems, estimating that roughly 30% of tasks are broken. The company has withdrawn its earlier recommendation to report the benchmark and says coding evaluations need stronger validation.
OpenAI says it has withdrawn its recommendation to use SWE-Bench Pro after an internal audit found widespread task-quality problems in the coding benchmark.
In its post, “Separating signal from noise in coding evaluations,” OpenAI says its review estimated that about 30% of SWE-Bench Pro tasks are broken. The company says these issues make the benchmark unreliable for evaluating coding systems without additional scrutiny.
The Stack reported that OpenAI had previously recommended SWE-Bench Pro as a replacement for SWE-bench Verified, but has now retracted that recommendation after finding that nearly 30% of tests were broken. The Decoder separately reported that OpenAI pulled its endorsement after its review found roughly the same level of flawed tasks.
According to OpenAI’s post, the problems were not limited to minor formatting errors. The company says broken tasks can include cases where tests do not correctly represent the intended software change, where solutions are ambiguous, or where evaluation logic can reward behavior that does not actually fix the underlying issue.
OpenAI argues that these flaws matter because coding benchmarks are increasingly used to compare frontier AI systems. If a benchmark contains many defective tasks, reported scores may reflect quirks in the test set rather than real software-engineering ability.
OpenAI’s earlier post, “Why SWE-bench Verified no longer measures frontier coding capabilities,” explains the context for its initial recommendation. The company said SWE-bench Verified had become less useful for distinguishing top coding models, and in February 2026 recommended reporting SWE-Bench Pro instead. The new audit reverses that position.
The Stack described OpenAI’s move as a call for new AI coding benchmarks to replace tests it now considers unreliable. The publication reported that OpenAI no longer recommends SWE-Bench Pro after identifying widespread defects.
The Decoder also reported that Artificial Analysis dropped SWE-Bench Pro from its rankings following the findings. The Decoder characterized the issue as a significant setback for a popular coding evaluation at a time when model developers and customers are looking for dependable ways to measure coding performance.
Coding benchmarks are used by AI labs, independent evaluators, and customers to compare systems on tasks that resemble real software maintenance. A task may ask a model to inspect a codebase, identify a bug, make a change, and pass tests. In principle, that format can be more realistic than short programming puzzles.
But OpenAI’s audit highlights a central risk: if the task itself is wrong, a high score can be misleading. A model may pass a defective test without producing the intended fix, or fail a task because the reference expectation is incorrect.
OpenAI says stronger benchmark construction and auditing are needed, including clearer task definitions and better validation of test behavior. The company’s conclusion is narrower than a claim about any specific model’s capability: it is warning that one benchmark’s task quality is not strong enough to support confident comparisons.
For AI buyers and researchers, the practical takeaway is caution. Scores on coding benchmarks can be useful, but OpenAI’s findings show they should be read alongside details about task validation, error rates, and independent audits rather than treated as standalone proof of coding ability.
OpenAI withdraws support for SWE Bench Pro OpenAI says it has withdrawn its recommendation to use SWE Bench Pro after an internal audit found widespread task quality problems in the coding benchmark.
In its post, “Separating signal from noise in coding evaluations,” OpenAI says its review estimated that about 30% of SWE Bench Pro tasks are broken.
The company says these issues make the benchmark unreliable for evaluating coding systems without additional scrutiny.
Continue reading
