What it does

LiteLLM is an open source AI gateway that gives one interface to more than 100 LLM providers, including OpenAI, Anthropic, Gemini, Bedrock and Azure, using the OpenAI format. Supported endpoints include chat completions, responses, embeddings, images, audio, batches and rerank. You can use it in two ways. As a Python SDK it sits inside your code, with a Router that adds retries and fallbacks across deployments, application-level load balancing, cost tracking and observability callbacks. As a proxy server it runs as a central service for a team, with virtual keys, per-project and per-user spend tracking, guardrails, caching, logging and an admin dashboard. It can also bridge MCP servers to any model, invoke agents over the A2A protocol, and run coding-agent harnesses on a model of your choice through the gateway. Docker images and Terraform modules for AWS and GCP are described for deployment. A separate Enterprise tier adds single sign-on, custom integrations, custom SLAs and professional support; prices are not given on the page.

Capabilities

One OpenAI-format interface to 100+ LLM providers
Virtual keys, multi-tenant spend tracking and an admin dashboard
Router with retries, fallbacks and load balancing
Guardrails, caching and logging per project
MCP gateway and A2A agent invocation
Terraform modules for deploying on AWS or GCP

When to use it

Giving a team one gateway to many model providers
Tracking spend per project or user across providers
Switching providers in Python code without rewriting calls
Exposing MCP tools to models through a central service