ICAE-Bench is an interactive benchmark for coding agents that treats requirement clarification as part of project-building performance. The supplied arXiv, GitHub, and Hugging Face sources describe 480 anonymized tasks across 12 programming languages, an automated interactive Oracle, and evaluation tooling spanning black-box tests plus functional, structural, design, agentic, and interaction-oriented diagnostics....
ICAE-Bench is presented as a benchmark for assessing coding agents as interactive project builders, rather than solely as systems that generate code from a fixed specification. The canonical arXiv record describes a setting in which agents must clarify fuzzy product requirements with an automated user agent before or while constructing a software project. This reframes requirement elicitation as an evaluated capability: the benchmark is not limited to whether an agent writes code that appears plausible from an initial prompt, but includes whether it can seek information necessary to resolve ambiguity.
The official GitHub repository, ALEX-nlp/ICAE-EVAL, states that ICAE-Bench contains 480 anonymized tasks spanning 12 programming languages. It also identifies an interactive Oracle and tooling for tests, structural assessment, agentic evaluation, and interaction-quality assessment. The supplied Hugging Face Papers record corroborates the 480-task, 12-language characterization and links to the benchmark dataset and evaluation-code repository.
The available sources therefore support a narrow but important claim: ICAE-Bench operationalizes coding-agent evaluation around an interaction loop between an agent and a benchmark-controlled user surrogate. Its stated assessment dimensions extend beyond externally observable functionality to include functional, structural, design, and interaction diagnostics. The sources do not specify how these dimensions are combined, whether they form a composite leaderboard metric, or whether some are intended only as diagnostic outputs.
The distinguishing mechanism named across the supplied materials is the interactive Oracle. According to the arXiv excerpt, coding agents clarify fuzzy product requirements through an automated user agent; the GitHub repository calls this component an interactive Oracle. This implies a task interface in which agents may request clarification instead of being forced to infer every missing requirement from an initial natural-language description.
Such an interface changes the target behavior under evaluation. In a conventional fixed-prompt code-generation task, all relevant requirements are assumed to be already available, whether explicitly or implicitly. In ICAE-Bench’s stated setup, useful behavior can include recognizing underspecification, formulating a targeted follow-up question, interpreting the response, and applying the resolved constraint while building the resulting project.
The repository excerpt also names evaluation tooling for tests, structural assessment, agentic evaluation, and interaction quality. The arXiv excerpt explicitly mentions black-box tests. In the supplied evidence, black-box testing is the clearest defined assessment category: it concerns externally observable behavior rather than direct inspection of internal implementation choices. The meanings of structural assessment, design diagnostics, agentic evaluation, and interaction-quality evaluation are not specified in the supplied sources. Their rubrics, implementation methods, and validity procedures should therefore not be inferred.
Critical implementation details of the Oracle are also absent from the provided source excerpts. The materials do not state whether Oracle responses are scripted, generated by a language model, retrieved from task metadata, or produced through a hybrid mechanism. They do not specify response consistency controls, prompting templates, state management, turn limits, answer-validation procedures, determinism settings, or safeguards against contradictory answers. The sources also do not say whether agents can revise project artifacts after receiving clarifications.
The dataset composition supported by the supplied records is limited to 480 anonymized tasks across 12 programming languages. The identities of those languages, the number of tasks per language, task domains, repository sizes, dependency profiles, and file counts are not specified. Neither are the origins of the tasks: the supplied excerpts do not state whether examples were authored specifically for the benchmark, adapted from existing projects, derived from issue trackers, or synthesized.
The benchmark’s described protocol combines black-box tests with functional, structural, design, and interaction diagnostics. This is evidence of a multi-axis assessment design, but not of a fully specified scoring system. The sources do not provide task-level rubrics, pass conditions, metric scales, scoring weights, evaluator prompts, human-review procedures, automated judges, or calibration results. It is also not specified whether black-box test success is a prerequisite for other evaluations or whether the diagnostic dimensions can diverge—for example, whether an agent can be rated highly on interaction quality while delivering an incorrect artifact.
No evaluated systems or baselines are named in the supplied sources. The excerpts do not identify proprietary models, open-weight coding models, agent frameworks, retrieval components, tool-use environments, prompting strategies, or scaffolding approaches. They further do not specify whether agents receive terminal access, package-installation privileges, network connectivity, browser tools, test-running access, source-control access, or fixed token and compute budgets.
Data-split design is similarly unspecified. The supplied sources do not describe development, validation, or held-out test partitions; hidden-test policies; contamination controls; duplicate detection; task-release sequencing; or whether all 480 tasks are publicly accessible. The repository is described as releasing the benchmark and associated evaluation tooling, but exact access conditions and dataset packaging are not established by the excerpts.
The supplied arXiv, GitHub, and Hugging Face excerpts do not report numerical results. There are no supported claims about pass rates, per-language performance, functional scores, structural scores, interaction scores, confidence intervals, rank orderings, ablations, or model-specific failure rates. No comparative conclusion about coding-agent capability can be drawn from the provided evidence.
The only supported metric-level characterization is categorical. ICAE-Bench assesses agents using black-box tests and diagnostics described as functional, structural, design, and interaction-oriented. The sources do not specify whether these outputs are binary, continuous, rubric-based, model-judged, programmatically computed, or human-validated. They also do not identify a primary metric or explain how reported benchmark performance should be interpreted.
This distinction matters for technical use. Multi-dimensional evaluation can expose differences that a single test-pass measure misses, especially when a system generates an artifact that behaves correctly in narrow cases but handles interaction or project organization poorly. But the supplied sources do not establish measurement reliability, metric correlation, robustness to evaluator variation, or whether the dimensions predict performance on real software-engineering workflows.
The official repository listed in the supplied materials is ALEX-nlp/ICAE-EVAL. Its description states that it releases ICAE-Bench, the interactive Oracle, and tooling for tests, structural assessment, agentic evaluation, and interaction quality. The supplied Hugging Face Papers page independently links readers to the benchmark dataset and evaluation-code repository.
These links establish that implementation-oriented resources accompany the benchmark description. However, the supplied source excerpts do not specify a software license, dataset license, release version, installation procedure, dependency lockfile, container configuration, hardware requirements, random-seed policy, evaluator-model version, or command-line execution interface. They also do not establish whether reference implementations, hidden tests, Oracle configurations, evaluator prompts, or all task metadata are public.
The canonical paper link supplied is arXiv:2607.21217. Paper authors and institutional affiliations are not included in the supplied record and are therefore not specified here.
The main limitation is evidentiary. The supplied materials provide a concise benchmark description and repository summary rather than full methodological detail, experimental tables, or appendices. As a result, claims about benchmark difficulty, interaction realism, Oracle behavior, annotation quality, evaluator reliability, and comparative model performance would be unsupported.
The interactive Oracle is central to the benchmark’s framing, but its behavior can substantially influence task difficulty. If Oracle responses are inconsistent, overly revealing, or sensitive to wording, agents may be rewarded or penalized for properties unrelated to software-project construction. The supplied excerpts do not provide enough detail to assess those risks.
Likewise, the tasks are described as anonymized, but the materials do not explain what was anonymized or how anonymization affects provenance, contamination analysis, and external validity. The sources do not describe task sourcing, reference-solution construction, test coverage, ambiguity design, or whether all tasks require interaction. These omissions should constrain interpretation until the full paper and repository materials are examined.
ICAE-Bench’s stated focus addresses a gap between fixed-specification coding tests and practical project work, where initial product requirements may be incomplete. The benchmark’s contribution is methodological: it makes clarification behavior visible to evaluation rather than treating it as outside the coding task.
For builders of coding-agent systems, the relevant implication is not that any particular model has demonstrated superior performance—no such results are included in the supplied sources. Instead, it is that evaluation can consider both the delivered software artifact and the interaction used to build it. Black-box behavior can test whether a project works externally, while interaction-focused diagnostics may assess whether an agent asked appropriate questions under uncertainty.
Teams considering ICAE-Bench should inspect the full paper and released repository before using it for model selection, capability claims, or release gating. In particular, they should verify task provenance, Oracle behavior, tool permissions, scoring rules, hidden-test policy, and the reproducibility of structural, design, agentic, and interaction-quality assessments.
Published Jul 23, 2026, 12:00 AM
By Kainotomic Team
Published Jul 23, 2026, 12:00 AM
