Nutrient has released agentic-usability, an open source CLI for evaluating how well AI coding agents can work with software development kits. The project generates SDK programming tasks, runs agents in isolated sandboxes, and scores results across discovery, correctness, completeness, and execution.
Nutrient has open sourced agentic-usability, a command-line tool designed to measure how well AI coding agents can use a software development kit.
In a blog post titled “Introducing agentic-usability: Measuring how well AI agents can use your SDK,” Nutrient describes the project as a way to test whether coding agents can understand and apply an SDK in realistic programming tasks. The company says the tool generates SDK programming challenges, runs AI coding agents in microVM sandboxes, and evaluates the resulting work with an LLM-based judge.
The GitHub README for PSPDFKit-labs/agentic-usability describes Agentic Usability as a CLI tool for measuring how well AI coding agents such as Claude Code, Codex, and Gemini CLI can use an SDK. That framing places the project in a growing category of developer tooling focused not just on model performance in isolation, but on how agents interact with real-world developer interfaces, documentation, and APIs.
According to Nutrient, agentic-usability creates programming challenges for an SDK and then runs coding agents against those tasks in isolated microVM environments. The use of sandboxes is intended to let agents attempt functional implementations without relying only on static analysis or written answers.
Nutrient says the tool scores results using an LLM judge across several dimensions: API discovery, correctness, completeness, and functional execution. Those categories reflect common failure points for agentic coding systems. An agent may find the right API but use it incorrectly, produce incomplete code, or generate an implementation that appears plausible but fails when executed.
The project’s focus on functional execution is notable because SDK usability often depends on details that are hard to capture in documentation reviews alone. If an agent cannot install, import, configure, or call an SDK successfully, the issue may lie in documentation clarity, API design, examples, packaging, or the agent’s own reasoning limits.
The GitHub releases page for PSPDFKit-labs/agentic-usability lists version 0.2.0 as the latest release and version 0.1.0 as the initial release. That indicates the project is still early, but already has public release artifacts for developers who want to inspect or run it.
Nutrient’s blog post says the tool is open sourced, and the public GitHub repository provides the README and release history. The available source descriptions do not specify a universal benchmark score, a leaderboard, or comparative results across SDKs, so the release should be understood primarily as a tool for SDK maintainers to run their own evaluations rather than as a published ranking of coding agents.
For SDK publishers, agentic-usability targets a practical question: can AI coding tools successfully use the SDK without human intervention? As developers increasingly rely on coding assistants and agentic tools, SDKs may need to be legible not only to people but also to automated systems that search documentation, infer API usage, write code, and test outputs.
A tool like agentic-usability could help teams identify where agents get stuck. For example, low scores in API discovery could suggest that names, examples, or documentation structure are difficult for agents to navigate. Failures in functional execution could point to setup complexity, missing examples, ambiguous instructions, or brittle integration paths.
Nutrient’s release does not claim to solve SDK usability on its own. Instead, based on the company’s description and the GitHub README, it provides a structured way to generate tasks, run agents, and score their attempts. That makes it a diagnostic tool for teams that want to understand how their SDKs perform in agent-assisted development workflows.
PSPDFKit-labs/agentic-usability READMEPSPDFKit-labs/agentic-usability releases pageNutrient has open sourced agentic usability, a command line tool designed to measure how well AI coding agents can use a software development kit.
The company says the tool generates SDK programming challenges, runs AI coding agents in microVM sandboxes, and evaluates the resulting work with an LLM based judge.
The GitHub README for PSPDFKit labs/agentic usability describes Agentic Usability as a CLI tool for measuring how well AI coding agents such as Claude Code, Codex, and Gemini CLI can use an SDK.
Continue reading