LiteLLM

LiteLLM is an open-source AI gateway that lets you call 100+ large-language-model providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, Groq, Mistral, Ollama and more — through a single OpenAI-compatible interface, and layers on virtual keys, spend tracking, guardrails, fallbacks, and observability. We rate it 84/100 — it is the default abstraction layer for teams that want to stay multi-model without rewriting their stack every time a new frontier model ships, and it scales from a one-line Python SDK import to a production proxy deployed by Stripe, Netflix and Google.

What is LiteLLM?

LiteLLM is built by BerriAI, a Y Combinator W23 company founded by Ishaan Jaffer and Krrish Dholakia. The first public release landed in 2023 as a thin Python shim that normalized OpenAI, Anthropic and Cohere SDKs, and the project has since grown into a full-blown AI gateway with an admin UI, virtual-key management, budgets, logging pipelines, and MCP and A2A protocol bridges. As of the latest release v1.83.7-stable on April 19, 2026, the repository has 43.9k GitHub stars, 7.4k forks, and 1,438 contributors. OSS adopters publicly listed in the repo include Stripe, Netflix, Google ADK, OpenAI Agents SDK, Greptile and OpenHands.

The specific problem LiteLLM solves is multi-provider fragmentation. Every LLM vendor ships its own SDK, its own authentication pattern, its own request and response shape, and its own error taxonomy. Swap providers for cost, latency or reliability reasons and you are rewriting glue code. LiteLLM collapses all of that into a single OpenAI-shaped completion() call — or, if you deploy the proxy, a single base_url your team can point at from any language. That one indirection is what makes multi-model strategies tractable.

Key Features of LiteLLM

• 100+ LLM providers, one API: OpenAI, Anthropic, Azure OpenAI, Vertex AI, Bedrock, Gemini, Cohere, Mistral, Groq, Together AI, HuggingFace, vLLM, Ollama, Replicate and 85+ more — all callable through the same /chat/completions, /responses, /embeddings, /images, /audio, /rerank and /messages endpoints.
• Production-grade proxy: Docker-deployable AI Gateway with ~8 ms P95 overhead at 1k RPS according to the official benchmarks, plus load balancing, automatic retries, and cross-provider fallbacks (fail over from Anthropic → OpenAI → Azure on a single 429).
• Virtual keys, budgets and spend tracking: Issue scoped keys per team, user, or project, attach monthly or daily budgets, and get per-request cost attribution down to the model — with optional S3, GCS, Langfuse, Datadog or Helicone log sinks.
• Guardrails and PII redaction: Plug in Lakera, Presidio, Aporia, Bedrock Guardrails or custom hooks to scrub PII and block unsafe outputs before they leave the proxy.
• MCP and A2A gateway: Act as a Model Context Protocol and Agent-to-Agent protocol broker, bridging agents built on LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore and Pydantic AI to any model behind the proxy.
• Admin dashboard: A React UI for creating keys, viewing usage, setting rate limits and watching live traffic — no SQL or kubectl required.
• Drop-in compatibility: Point any OpenAI SDK (openai, LangChain, LlamaIndex, Instructor) at http://localhost:4000 and it just works — no code changes needed.

What Users Say About LiteLLM

On Hacker News, the most-upvoted thread captures the split sentiment well: developers praise how quickly LiteLLM exposes new models — several teams report being able to offer a just-released frontier model to their users on launch day without any code changes — but also flag that the proxy is “kind of a mess TBH” when you run it at scale. The most recurring complaints across Reddit's r/LocalLLaMA and r/MachineLearning are (1) gradual memory growth that requires periodic worker recycling, (2) a roughly 500 µs fixed overhead per request that some teams cite as painful for short completions, and (3) debugging requests through a proxy layer when a provider returns a weird shape. On Product Hunt the Python SDK has a near-perfect rating; the pain points cluster around the self-hosted proxy deployment, not the library itself.

LiteLLM Pricing

LiteLLM is free and open source under the MIT license for the SDK and a permissive license for the community proxy. The commercial offering is an Enterprise tier aimed at regulated and large-scale deployments.

Plan	Price	Key Limits
Open Source	$0	100+ provider integrations, logging, load balancing, guardrails, virtual keys, unlimited self-hosted usage
Enterprise	Custom (contact sales)	Adds JWT auth, SSO/SAML, SOC 2 audit logs, managed upgrades, priority support, SLA
Hosted Cloud	Usage-based	BerriAI-managed proxy for teams that don't want to run infra

There are no per-seat fees on the open-source tier and no cap on request volume — the only costs you pay are to the underlying model providers you route through LiteLLM.

Who Should Use LiteLLM?

Best for: Backend and ML platform teams who already make more than one LLM call a second, want a single internal API for multiple providers, and need per-team budgets and audit logs. Especially strong for regulated industries (finance, healthcare) that need a self-hosted proxy with no data leaving their VPC.

Not ideal for: Solo developers building a prototype — in that case the litellm Python SDK alone is enough, the full proxy is overkill. Also skip it if you're happy being all-in on one provider; there's no value in an abstraction you never use.

Pros and Cons

Pros:

• Switch providers with a one-line config change — no code rewrite.
• Cross-provider fallbacks let you survive an Anthropic 529 or an OpenAI region outage without paging anyone.
• Per-key, per-team, per-project spend visibility that Langfuse-style tools can slot into via native hooks.
• New frontier models are usually added within 24 hours of their public launch — several users report shipping day-one support through LiteLLM.
• Truly open source and MIT-licensed — you can fork it, audit it, and run it in an air-gapped VPC.

Cons:

• Proxy memory footprint grows over time; scheduled worker recycling is a common operational pattern.
• Roughly 500 µs of fixed overhead per request — meaningful for latency-sensitive chat apps.
• Debugging a failing request through the proxy is harder than calling the upstream provider directly.
• Running the full stack needs Redis and Postgres — not a "just one binary" experience like some competitors.

Alternatives to LiteLLM

OpenRouter offers a managed, hosted alternative with no self-hosting required — simpler, but you lose data control and self-hosted audit trails. Langfuse is complementary rather than competitive — it handles tracing and evals while LiteLLM handles routing, and the two are commonly paired. Portkey offers a closed-source SaaS gateway with a nicer UI but a less permissive licence.

Verdict: Is LiteLLM Worth It?

For any team that has moved past the prototype stage and is making LLM calls against more than one provider, LiteLLM is the default choice — it is what Stripe, Netflix and Google build on top of, and it has more frontier-model coverage than any managed alternative. The rough edges (memory growth, 500 µs overhead, operational footprint) are real but well-documented, and the Enterprise tier exists precisely to carry those for you. Our 84/100 reflects a tool that is best-in-class at its core job but still requires platform-engineering maturity to run in production.

Frequently Asked Questions

Is LiteLLM free?

Yes. The LiteLLM SDK and community proxy are free and open source under the MIT license, with unlimited self-hosted usage. You only pay the underlying LLM providers. Enterprise pricing for JWT auth, SSO and SLAs is custom and negotiated with BerriAI sales.

What LLM providers does LiteLLM support?

100+ providers including OpenAI, Anthropic, Azure OpenAI, Google Vertex AI, AWS Bedrock, Gemini, Cohere, Mistral, Groq, Together AI, HuggingFace, vLLM, Ollama and Replicate — all callable through the same OpenAI-format API.

How does LiteLLM compare to OpenRouter?

OpenRouter is a managed SaaS gateway — zero infra, but your traffic is routed through a third party. LiteLLM is an open-source gateway you self-host, which keeps data inside your VPC and lets you bring your own provider keys. Teams with compliance requirements pick LiteLLM; teams that just want a single API key pick OpenRouter.

Is LiteLLM open source?

Yes — the SDK and community proxy are MIT-licensed on GitHub. Enterprise-only features (JWT auth, SSO, audit logs) are under a separate commercial licence.

Can LiteLLM track costs per user or team?

Yes. Virtual keys can be scoped to teams, users, or projects, each with its own monthly and daily budget, and every request is logged with the upstream cost — visible in the admin UI or exportable to S3, GCS, Langfuse or Datadog.