Anthropic Outlines How Teams Should Evaluate AI Agents Before Deployment

Anthropic has published an engineering guide explaining how teams can evaluate AI agents more systematically before deploying changes or upgrading models.

The post, titled “Demystifying evals for AI agents,” defines agent evaluations as structured tests that include tasks, trials, graders, traces, outcomes, and the harness that runs the evaluation. Anthropic says automated evaluations can be used during development and in CI/CD workflows when teams modify an agent or change the underlying model.

Why agent evaluation is broader than answer checking

Traditional model evaluation often focuses on whether a final answer is correct. Agent evaluation has to inspect more of the workflow because agents may call tools, retrieve data, write code, select APIs, or take several steps before producing a response.

Braintrust makes a similar distinction in its article “What is agent evaluation?,” describing agent evaluation as workflow-level testing. It says teams should assess tool selection, parameter accuracy, intermediate outputs, and task completion, rather than relying only on a final response score.

That distinction is important because a plausible final answer can hide an unreliable process. Anthropic’s framework includes traces, which allow teams to inspect the sequence of actions that produced an outcome. This supports trajectory-style evaluation: checking not only what the agent answered, but how it arrived there.

Golden datasets help catch regressions

Anthropic, Braintrust, LangSmith, and OpenAI all point toward the value of stable evaluation sets. LangSmith’s “Evaluate a chatbot” tutorial describes creating an initial golden dataset, defining metrics, comparing prompts or models, tracking results over time, and automating evaluations in CI/CD.

OpenAI describes a related practice in “Inside OpenAI’s in-house data agent.” The company says its internal data agent uses curated question-and-answer evaluation sets with manually authored golden SQL. OpenAI says these development-time evaluations help catch regressions before user trust is damaged.

A golden dataset gives teams a consistent baseline. When a prompt, model, retrieval component, or tool changes, the same cases can be replayed and compared. If a system fails tasks it previously handled, teams have concrete evidence of a regression instead of relying on impressions from ad hoc testing.

Graders, rubrics, and repeated trials

Anthropic’s guide separates the task being tested from the grader that scores performance. A grader can be deterministic, such as an exact-match check or programmatic validation, or more flexible, such as an LLM-based judge using a written rubric.

Rubric-based grading is useful when there is no single correct string. For example, a support agent’s response may need to be accurate, concise, polite, and safe. Scoring those dimensions separately can show whether a change improved factual accuracy while harming tone, or reduced cost while increasing errors.

Braintrust also emphasizes repeated trials. Agent systems can be nondeterministic, meaning the same input may lead to different tool calls, intermediate steps, or final responses across runs. Multiple trials help reveal whether an agent is consistently reliable or only occasionally succeeds.

Tool behavior needs its own tests

Agent failures are not always model failures. Braintrust identifies tool selection and parameter accuracy as evaluation targets, while Anthropic’s component-based framing includes traces and the surrounding evaluation harness.

In practice, teams may need to test tools directly as well as test the full agent. If an API wrapper, search function, file parser, or database query is broken, the agent can fail even when the model’s reasoning is adequate. Separating tool-level tests from end-to-end agent evaluations can make failures easier to diagnose.

Offline testing is not enough

Offline evaluations help teams compare versions before release, but production monitoring remains necessary. Braintrust includes monitoring among agent evaluation practices, and LangSmith describes tracking evaluation results over time.

The limitation is that curated test sets may not fully match real user behavior. Users may phrase requests differently, provide unexpected data, or trigger edge cases that were missing from the evaluation set. For higher-risk changes, teams can compare candidate systems before a broad rollout, and live experiments may be useful when offline scores no longer predict user outcomes.

The practical takeaway

Across Anthropic, Braintrust, LangSmith, and OpenAI, the shared message is that agent evaluation should be structured, continuous, and specific. A useful evaluation program includes stable datasets, explicit success criteria, graders, trace inspection, regression checks, repeated trials, and monitoring after deployment.

The strongest tests are not necessarily the most complex. They are the tests that would have caught the last serious failure: a bad tool call, a prompt regression, an unsafe response, a brittle workflow, or a model upgrade that changed behavior in an unexpected way.