← Back to Discover

skill-optimizer

Docker-based workbench and agent skill for running deterministic evals on agent skills, with a CLI that runs cases and suites against OpenRouter models and grades the results.

Observability & evalsCommand-line tools
Visit website ↗

What it does

skill-optimizer is an open-source repository with two uses. You can install its skill or plugin into an agent (Claude Code, OpenAI Codex, Cursor, OpenCode or Gemini CLI) so the agent can author and debug eval suites, or you can run its local CLI to execute eval cases and suites inside Docker against OpenRouter models. The workbench gives the agent under test a skill or reference folder and an isolated work directory. A case is a user-like task with one or more graders, and a suite is a matrix of cases and models. Graders run after the agent and stay hidden from it; they can inspect workspace files, artifacts, the answer, a trace and other local state, which suits evals whose success can be checked deterministically. Examples in the repository include a PDF suite with extraction, splitting and creation cases plus a negative case, and an MCP suite built around a local calculator server. Real model runs need Node.js 20+, Docker and an OpenRouter API key, and only openrouter/ model references are supported.

Capabilities

  • Skill and plugin for Claude Code, Codex, Cursor, OpenCode and Gemini CLI
  • run-case and run-suite CLI commands for evals
  • Isolated Docker workbench with graders hidden from the agent
  • Deterministic grading from files, command logs, SQL and artifacts
  • Suites that combine cases with OpenRouter models

When to use it

  • Testing whether an agent skill holds up across several models
  • Writing evals where success can be verified from files or logs
  • Checking that an agent does not load a skill for an unrelated task