A new arXiv paper and reference GitHub repository introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark for comparing OpenClaw-style coding-agent harnesses under shared prompts, runtime limits, workspace rules, patch collection, evaluation, and cost accounting.
The Claw-SWE-Bench authors introduced a benchmark and adapter protocol for evaluating OpenClaw-style coding-agent harnesses on software engineering tasks, according to the paper listed on arXiv and Hugging Face Papers.
Claw-SWE-Bench is presented on arXiv as a multilingual SWE-bench-style benchmark for comparing OpenClaw-style coding-agent harnesses across 350 tasks, with an 80-instance Lite subset for smaller-scale evaluation. The Hugging Face Papers listing describes the project as standardizing fixed prompts, runtime budgets, workspace contracts, patch extraction, evaluators, and cost accounting.
That focus is notable because coding-agent results can vary not only by the underlying language model, but also by the surrounding harness: how the task is presented, what files and tools are available, how long the agent can run, how patches are collected, and how outputs are judged. By defining those pieces explicitly, Claw-SWE-Bench aims to make comparisons between supported OpenClaw-style harnesses more consistent.
The GitHub repository for opensquilla/claw-swe-bench describes itself as the reference implementation for Claw-SWE-Bench. According to the repository excerpt, it supports unified evaluation of agent harnesses using identical prompting, patch collection, and SWE-bench evaluation.
The Hugging Face Papers summary adds that Claw-SWE-Bench standardizes runtime budgets, workspace contracts, patch extraction, evaluators, and cost accounting. In practice, those categories address common sources of variation in coding-agent benchmarks: whether one system received more time, a different prompt, a different working directory layout, or a different method for converting its changes into a patch.
The arXiv description also says the benchmark includes an adapter protocol. That suggests the project is intended to make different OpenClaw-style harnesses conform to a shared interface for evaluation rather than requiring each harness to be measured with bespoke scripts.
According to the arXiv excerpt, Claw-SWE-Bench covers 350 tasks and includes an 80-instance Lite subset. The paper describes the benchmark as multilingual, extending the SWE-bench-style approach beyond a single-language framing.
The existence of a Lite subset may be useful for researchers or developers who want faster iteration before running the larger benchmark. However, the provided sources do not report comparative performance results, rankings, or claims that one harness outperforms another. The available descriptions focus on the benchmark’s structure and reference implementation rather than on leaderboard outcomes.
SWE-bench-style tests have become a common way to evaluate systems that attempt to solve real software issues by editing code and producing patches. For agentic coding systems, the harness can strongly influence results because it governs tool use, prompt format, execution limits, and how final changes are submitted.
Claw-SWE-Bench’s stated contribution is to make those variables explicit for OpenClaw-style systems. The Hugging Face Papers listing specifically cites fixed prompts and runtime budgets, while the GitHub repository highlights identical prompting, patch collection, and SWE-bench evaluation. Together, the sources describe a benchmark intended to reduce ambiguity in comparisons between harnesses.
The benchmark is listed on Hugging Face Papers under the title “Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks.” The corresponding arXiv entry provides the paper description, and the opensquilla/claw-swe-bench GitHub repository is described as the reference implementation.
For now, the source material supports a limited but clear conclusion: Claw-SWE-Bench is a newly presented benchmark package for evaluating OpenClaw-style coding-agent harnesses with shared task definitions, adapter conventions, prompting, patch handling, evaluation, and accounting controls.
The Claw SWE Bench authors introduced a benchmark and adapter protocol for evaluating OpenClaw style coding agent harnesses on software engineering tasks, according to the paper listed on arXiv and Hugging Face Papers.
The Hugging Face Papers listing describes the project as standardizing fixed prompts, runtime budgets, workspace contracts, patch extraction, evaluators, and cost accounting.
By defining those pieces explicitly, Claw SWE Bench aims to make comparisons between supported OpenClaw style harnesses more consistent.
Continue reading
