What it does
Opik by Comet logs each step an agent takes, from user input and context retrieval to tool calls, so you can inspect traces, have subject-matter experts annotate weak ones, and see where failures occur. It is meant to cover development, testing and production. Evaluation includes LLM-as-a-judge scoring with more than 30 metrics such as answer relevance, hallucination and task completion, plus Test Suites where you write plain-text assertions that give pass or fail results. In production it can evaluate traces in real time, alert on interactions that fail your criteria, apply guardrails against policy violations and PII exposure, and track token usage and model cost. Opik also includes Ollie, an assistant that analyzes traces and writes fixes into your code along with a regression test, an Agent Playground for running a whole agent with different models, prompts and parameters, a Prompt Optimizer with six algorithms, and Cost Intelligence for tracking Claude Code and Codex usage across engineering teams. The core observability and evaluation features are open source and can be run locally from the GitHub code, and Comet also offers an enterprise version. The page mentions a free tier that needs no credit card.
Capabilities
- Tracing of agent steps, retrieval and tool calls
- LLM-as-a-judge evaluation with 30+ metrics
- Test Suites with plain-text pass or fail assertions
- Ollie assistant that proposes code fixes with regression tests
- Agent Playground and Prompt Optimizer
- Production monitoring, guardrails and cost tracking
When to use it
- You build LLM apps or agents and need traces and evaluations across development and production
- You want an open-source tool you can run locally before moving to a hosted or enterprise setup
- You want to track what coding agents such as Claude Code and Codex cost your team
Alternatives to explore
- LangSmith
- Langfuse
- Arize
- Braintrust
- Galileo
