What it does
OpenLIT describes itself as an open-source platform for agent harness engineering. It records what AI agents and LLM applications do, including model calls, tool calls, retrieval and sub-agent steps, as OpenTelemetry traces and metrics, and shows them in a dashboard backed by ClickHouse. On top of tracing it offers evaluations (LLM-as-a-judge, programmatic checks and CI gates), runtime guardrails for issues such as prompt injection, a versioned Prompt Hub, a Rule Engine, a secrets Vault, and cost tracking with custom pricing. It also monitors NVIDIA, AMD and Intel GPUs used for inference, and OpenGround lets you run the same prompt across providers to compare cost, latency and output quality. You instrument an application with a one-line init call in the Python or TypeScript SDK, and a Go SDK is also listed. A separate CLI captures sessions from coding agents such as Claude Code, Cursor, Codex and OpenCode. Telemetry can also be sent to other OTLP backends such as Grafana or Datadog. The site says the core platform is free to self-host under Apache 2.0 using Docker or Kubernetes, and that custom roles, audit logs and alerts are part of OpenLIT Enterprise.
Capabilities
- OpenTelemetry traces and metrics for LLM calls, tools and retrieval
- Evaluations with LLM-as-a-judge checks and CI gates
- Guardrails, Prompt Hub, Rule Engine and secrets Vault
- Cost tracking and GPU monitoring
- Side-by-side model comparison in OpenGround
- Python, TypeScript and Go SDKs plus a coding-agent CLI
When to use it
- You run LLM apps or agents and want traces and costs kept in your own infrastructure
- You already use OpenTelemetry and want AI telemetry to flow into the same stack
- You want to see what a coding agent such as Claude Code or Codex did in a session
- You want eval checks on production traces or as CI gates
Alternatives to explore
- Langfuse
- LangSmith
- Arize Phoenix
- Braintrust
- Helicone
- Datadog
- Comet Opik
