Graphistry and Louie.ai Launch BotsBench to Benchmark AI Agents for SOC Investigations

Graphistry and Louie.ai have launched BotsBench, a benchmarking project for evaluating AI models and agents on security operations center investigations.

A benchmark for SOC-style AI investigations

BotsBench describes itself as a benchmark for AI models and agents used in SOC and incident-response investigations. According to the BotsBench site published by Graphistry and Louie.ai, the project evaluates systems on cybersecurity investigation tasks rather than general chat or coding prompts.

The benchmark is aimed at a practical question for security teams: how well can an AI model or agent work through the kind of evidence and reasoning required in an investigation?

Graphistry says in a LinkedIn post that Louie.ai autonomously solved Splunk Boss of the SOC and that BotsBench was launched to support vendor-neutral benchmarking, standardized BOTS-based runs, and leaderboards across models and agents. The company frames the work as a way to compare both model capability and agent harness design in a SOC context.

CyBT-CTF and BOTSv3

The BotsBench site says the project uses CyBT-CTF as a blinded benchmark and Splunk BOTSv3 as a public cross-check. That combination is important because public cybersecurity challenges can be useful for comparison but may also be present in training data or discussed in public writeups.

BotsBench explicitly references “contamination hygiene,” signaling an attempt to separate genuine investigative performance from memorization of public answers. The public BOTSv3 material gives outside observers a recognizable point of reference, while the blinded CyBT-CTF component is intended to make the evaluation harder to game.

A follow-up report by Leo Meyerovich of Graphistry describes BotsBench as a continuous evaluation of model, harness, and provider combinations on agentic cybersecurity investigations. That report says the evaluations include CyBT-CTF and Splunk BOTSv3 comparisons, and discusses GLM 5.2 as an open-source contender tested against systems from Anthropic and OpenAI.

Comparing models, harnesses, and providers

BotsBench is not presented only as a model leaderboard. Graphistry’s descriptions say the benchmark evaluates combinations of models, agent harnesses, and providers. That distinction matters because a SOC investigation system may depend not only on the underlying language model but also on how tools, prompts, retrieval, state management, and verification are organized around it.

The Ecosyste.ms Awesome catalog lists BotsBench as a live leaderboard by Graphistry for benchmarking AI agents such as Claude Code and OpenAI Codex on SOC investigation tasks using the Splunk BOTSv3 corpus. That independent listing supports the project’s positioning as a live comparison point for agentic cybersecurity systems.

Why it matters

Security operations work often involves stitching together logs, alerts, timelines, hypotheses, and evidence. Benchmarks that focus on that workflow can reveal different strengths and weaknesses than general-purpose exams or conversational tests.

The source material does not establish BotsBench as an industry standard, and the published descriptions come primarily from Graphistry, Louie.ai, and related materials. Still, the project adds to a growing effort to evaluate AI systems in domain-specific settings where correctness, traceability, and resistance to benchmark contamination are central concerns.

For buyers and security teams, the useful takeaway is not a single headline score. It is the emergence of more specific evaluation methods for AI-assisted SOC investigations, including blinded tests, public cross-checks, and comparisons across the full model-and-agent stack.