Anthropic Tests Whether LLMs Can Assemble Multi-Step Software Exploits

From vulnerability discovery to exploit chains

Anthropic says its Claude Mythos Preview demonstrated the ability to find complex software vulnerabilities, convert those findings into exploit primitives, and combine those primitives into complete end-to-end attack chains. The company described the results in its report, Measuring LLMs’ ability to develop exploits.

The evaluation addresses a harder cybersecurity question than whether a language model can identify a bug or explain a proof of concept. Developing an exploit can require a model to analyze a target program, derive useful technical primitives from a flaw, account for security mitigations, and connect multiple steps into a working sequence.

Anthropic’s description should not be read as evidence that a model can reliably compromise arbitrary real-world systems. Instead, it frames exploit development as a collection of capabilities that can be evaluated at increasing levels of difficulty. A system may perform well at vulnerability analysis, for example, but fail when asked to produce a stable exploit or combine separate techniques into an end-to-end chain.

Restricted access for defensive work

According to TechCrunch, Anthropic did not broadly release Mythos Preview. The company instead limited access to partners in its Project Glasswing initiative, with the stated purpose of defensive security work.

That limitation reflects the dual-use nature of exploit-development tools. Security teams may use exploit chains to establish the practical severity of a vulnerability, validate a patch, or test whether mitigations work. But the same technical capability could also make unpatched flaws easier to misuse.

Anthropic’s reported approach places its testing in a constrained security context, emphasizing both capability measurement and safeguards around access. The available information does not establish how consistently the model succeeds across different software targets or how its performance compares with human researchers.

ExploitBench proposes a capability ladder

A separate preprint from Carnegie Mellon University and Bugcrowd proposes a more structured framework for measuring this area. EXPLOITBENCH: A Capability Ladder Benchmark for LLM Cybersecurity Agents describes a 16-capability benchmark based on 41 hardened vulnerabilities in V8, the JavaScript engine used in Chromium-based browsers.

The benchmark uses deterministic oracles—predefined checks that can verify whether a task was completed—to measure progress from code coverage through arbitrary code execution. Its capability-ladder design is intended to distinguish between partial and complete success.

That distinction is important. A model could locate a vulnerable code path without achieving a practical exploit. It could also generate one useful primitive but fail to bypass a mitigation, make the result stable, or link several primitives into a functioning attack chain.

Taken together, Anthropic’s evaluation and the ExploitBench proposal show a growing focus on granular measurement of AI cyber capabilities. The relevant question is not only whether a model can discuss vulnerabilities, but which technical stages it can complete, under what constraints, and with what controls governing its deployment.