What it does
Opik is an observability and evaluation platform for applications built on language models, including RAG systems and multi-step agents. Teams add tracing with a Python or TypeScript SDK, or send OpenTelemetry data, and see each LLM call, tool run and retrieval step as a trace tree. The page lists integrations with dozens of frameworks and providers, among them LangChain, LlamaIndex, OpenAI, Anthropic, CrewAI and LiteLLM. For testing, Opik supports datasets and experiments, code-based and LLM-as-a-judge metrics (for example hallucination detection and RAG relevance), a prompt playground, and a PyTest integration so evaluations can run in CI. In production, dashboards track feedback scores, trace counts and token usage, and online evaluation rules help surface problems. An Agent Optimizer SDK and guardrails are also part of the project. The full platform, backend included, is Apache-2.0 licensed. It can be self-hosted with Docker Compose or Kubernetes and Helm, or used on Comet.com Cloud. A setup command also lets coding agents such as Claude Code and Cursor read traces and run evaluations through MCP.
Capabilities
- Traces LLM calls and multi-step agent runs, including tool calls and retrieval
- Datasets, experiments and LLM-as-a-judge evaluation metrics
- Prompt playground and prompt management
- Production dashboards and online evaluation rules
- PyTest integration for running evaluations in CI
- Apache-2.0 platform that can be self-hosted
When to use it
- Debugging why an agent or RAG pipeline gave a poor answer by inspecting its full trace
- Scoring prompt or model changes against a dataset before shipping
- Keeping LLM observability inside your own infrastructure
- Watching feedback scores and token usage of a production LLM application over time
Alternatives to explore
- LangSmith
- Arize Phoenix
- Langfuse
- Weights & Biases Weave
- Braintrust
