DataCurve publishes DeepSWE benchmark for long-horizon software-engineering agents

DataCurve has published DeepSWE, a long-horizon software-engineering benchmark designed to evaluate frontier coding agents on realistic repository-level tasks.

What DeepSWE measures

According to DataCurve’s DeepSWE site, the benchmark tracks coding-agent performance using metrics including Pass@1, average cost, completion time, and output-token counts. The public leaderboard, updated June 11, 2026, lists systems such as GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, Kimi K2.6, and other coding agents.

DataCurve describes DeepSWE as a benchmark for “long-horizon” software engineering rather than short coding puzzles. In its blog, DataCurve says the benchmark was published on May 26, 2026, and is built from original tasks spanning 91 repositories across five programming languages.

The official GitHub repository for datacurve-ai/deep-swe states that DeepSWE includes 113 tasks covering TypeScript, Go, Python, JavaScript, and Rust. The repository also says the benchmark uses isolated environments and program-based verifiers, a design intended to test whether submitted code changes produce the expected behavior rather than merely matching a static answer.

Why the design matters

Many coding benchmarks evaluate narrow tasks, such as completing a function or solving a single programming problem. DataCurve’s description positions DeepSWE differently: the benchmark asks agents to work in existing repositories, where success may require understanding project structure, dependencies, tests, and implementation details across multiple files.

The use of hand-written behavioral verifiers, described by DataCurve in its blog, is central to that approach. Instead of relying only on conventional unit tests or text comparison, these verifiers are intended to check whether an agent’s modification satisfies the behavioral requirement of the task. The GitHub repository similarly emphasizes program-based verification and isolated execution environments.

Those details are important for interpreting the leaderboard. Pass@1 indicates whether a model or agent solves a task on its first attempt under the benchmark conditions. Cost, time, and output-token measurements provide additional context about the resources required to reach an answer. A model that scores well but uses substantially more time or tokens may be evaluated differently from one that is faster or less expensive, depending on the use case.

What the sources do and do not establish

The available DataCurve sources establish the benchmark’s stated scope, task count, languages, repository count, verification approach, publication date, and leaderboard metrics. They also show that DataCurve is comparing contemporary coding agents using a common benchmark format.

The sources do not independently verify that DeepSWE is more predictive of real-world software-engineering productivity than other benchmarks. They also do not, by themselves, establish that any listed model is broadly superior for all coding work. As with other AI evaluations, results should be read in the context of the benchmark design, task selection, agent scaffolding, execution settings, and cost assumptions.

The broader takeaway

DeepSWE reflects a continuing shift in AI evaluation from isolated programming questions toward repository-level engineering tasks. By combining original tasks, multiple languages, isolated environments, and behavioral verification, DataCurve is attempting to measure whether coding agents can make working changes in realistic codebases.

For teams assessing AI coding tools, the most useful parts of DeepSWE may be less about a single ranking and more about the additional operational metrics. Pass rates, time, token usage, and cost together give a more complete picture of how coding agents perform under benchmarked conditions.