ToolsSurvey
Six LLM routers for multi-provider apps, ranked for routing, failover and cost
Six LLM routers compared for November 2026: how Bifrost, LiteLLM, OpenRouter, Vercel, Cloudflare and Portkey handle weighted routing, failover and cost across providers.

TL;DR
- Six LLM routers worth considering for multi-provider apps in November 2026, ranked for what they do on the request path.
- Bifrost ranks first: Apache 2.0, Go, 11 µs of added overhead per request at 5,000 RPS in its published benchmark, with weighted routing, retries and fallbacks sharing state with governance and audit.
- LiteLLM is the strongest Python-SDK router; OpenRouter is the fastest hosted path to a long provider list; Vercel and Cloudflare fit teams already on those platforms; Portkey fits teams that want hosted governance.
- The decisions that matter are self-hosted versus hosted, in-memory versus external state, and whether the router is also the governance point.
What an LLM router has to do
Multi-provider LLM apps have three failure modes. A provider rate-limits a key. A provider returns a 500 for a window. A newer model costs less than the current default for most prompts. The router’s job is to turn each of those into a configuration change instead of an incident.
A production router handles:
- Weighted routing. Primary to a fast provider, fallback to a secondary, optional tertiary. Weights tuned to provider pricing and latency.
- Retries and fallbacks. Short, bounded retry on retryable errors; fallback to the next provider on non-retryable failure or deadline miss.
- Health checks. Per-provider error-rate windows that temporarily lift a provider out of rotation and put it back automatically.
- Cost-aware model choice. Either explicit mapping (small vs large per route) or classifier-based complexity routing.
- Consistent surface. One OpenAI-compatible API so the application does not special-case each provider.
Most teams start with weighted plus retries and add the fancier bits later. The six routers here all cover the basics; they differ on runtime, state model and what else they bundle in the box.
How these six were scored
| Criterion | What was assessed |
|---|---|
| Runtime | Language, process model, scaling behaviour |
| Provider coverage | Number and quality of supported providers and models |
| Routing primitives | Weighted, fallback, retry, complexity-aware, model mapping |
| State | In-memory, Redis, Postgres, external control plane |
| Governance | Virtual keys, budgets, rate limits, model access rules |
| Observability | OpenTelemetry, Prometheus, trace content |
| Deployment | Self-hosted, hosted, hybrid, air-gapped, Edge |
Runtime and state decide infrastructure cost at scale; routing primitives decide whether the router stays useful as traffic grows.
1. Bifrost — self-hosted, in-memory, boring in the good way
Bifrost is an open-source AI gateway from Maxim AI, written in Go and released under Apache 2.0. It exposes one OpenAI-compatible API across 25+ providers and 10,000+ models. For routing, the useful details are in how it keeps state.
The retries and fallbacks guide documents weighted routing across keys and providers with per-provider retry and bounded fallback chains. Per-key rate limits, budgets and model access rules sit beside the routing config, so a change to provider weights and a change to a team’s budget are one restart apart.
Runtime cost is low enough to design against: the t3.xlarge benchmark reports about 11 µs of added overhead per request at 5,000 RPS with a 100 per cent success rate. Peak memory is roughly 3.3 GB of the instance’s 16 GB. State lives in memory after boot, with asynchronous writes to a database for persistence, so per-request path work is small.
The provider surface is wide: cloud providers (OpenAI, Anthropic, Bedrock, Vertex, Azure, Gemini, Groq, Mistral, Cohere, xAI) and self-hosted options (vLLM, SGLang, Ollama). Keeping OpenRouter as one provider behind Bifrost is a supported pattern for teams migrating off a hosted aggregator one model at a time.
Virtual keys tie routing to governance: each key carries its own budget and allowed-model list. Semantic caching deduplicates near-identical prompts with exact-match and similarity search. Observability is OpenTelemetry GenAI spans and Prometheus metrics from day one.
Deployment is a single binary or Docker image, Apache 2.0 at the core, from the source repository. It is the pick for teams who want one self-hosted router and one place to hang governance and observability.
2. LiteLLM — the Python SDK path
LiteLLM is a Python library and self-hosted proxy under MIT outside its enterprise directory. For routing, the routing guide covers simple shuffle, weighted round-robin, latency-based routing and fallbacks across a router object. Teams who already write their agents in Python frequently keep LiteLLM in-process as an SDK and add gateway features over time.
Scale behaviour is pod-count-driven: the production deployment guidance sizes the proxy at one Uvicorn worker per pod with 1 vCPU and 4 Gi of memory, plus Redis for shared limits and Postgres for spend. Capacity grows by adding pods; that is fine for mid-sized workloads and gets expensive at higher RPS.
LiteLLM is the right pick for Python-first teams who want a router they can also import as a library. Some teams keep LiteLLM as the SDK and put a faster gateway behind it; see the LiteLLM alternatives survey for the migration pattern.
3. OpenRouter — the fastest path to a long provider list
OpenRouter is a hosted aggregator that fronts hundreds of models through one OpenAI-compatible API. For a prototype or a small production app, it is the cheapest path to multi-provider access: one key, one base URL, no infrastructure.
The trade-offs are the usual hosted-aggregator ones. OpenRouter charges a fee when credits are bought (5.5 per cent on Standard, 8 per cent on Business, 5 per cent for crypto) and 5 per cent on bring-your-own-key usage above $25,000 of list-price inference a month. Requests pass through OpenRouter’s path; self-hosted is not an option. For teams moving off OpenRouter the common pattern is to keep it as one provider behind a gateway and switch models to direct provider keys one at a time — see the OpenRouter alternatives survey.
4. Vercel AI Gateway — Next.js-native, hosted
Vercel’s AI Gateway is a hosted proxy tuned for Next.js and the Vercel AI SDK. Routing covers primary/fallback across providers, unified billing and provider spend caps. For teams who already deploy on Vercel, enabling AI Gateway is a configuration change rather than a new piece of infrastructure.
The limits are the familiar hosted-gateway ones: requests pass through a third-party edge, spend caps are tied to Vercel billing and residency is Vercel’s. It is a strong pick for Vercel-first teams and an unusual pick outside that niche.
5. Cloudflare AI Gateway — hosted with unified billing
Cloudflare’s AI Gateway is a hosted proxy with caching, rate limiting, logging and Unified Billing across supported providers. Routing covers primary/fallback with retries; MCP Portals extends similar policy to MCP servers.
Cloudflare’s appeal is edge footprint and operational maturity: putting AI traffic through Cloudflare puts it on the same path as the rest of the stack for Cloudflare-first apps. Limits are the same hosted-gateway constraints as Vercel’s.
6. Portkey — hosted gateway with governance
Portkey is a hosted LLM gateway with provider fan-out, conditional routing, guardrails and governance. Conditional routing covers rule-based model choice per request, which overlaps with Bifrost’s complexity routing. For teams who want a hosted control plane but with more governance than OpenRouter, Portkey is the common pick.
The trade-off is runtime and residency: Portkey is a managed service, so self-hosted and air-gapped deployments mean paying for an enterprise plan or picking a different gateway. For teams who need self-hosting from day one, Bifrost is the direct replacement — see the Portkey alternatives survey.
Side-by-side
| Router | Runtime | Weighted | Fallback | Auto-routing | Governance | Deployment |
|---|---|---|---|---|---|---|
| Bifrost | Go, single binary | Yes | Yes, bounded | Complexity routing | Virtual keys, budgets, rate limits | Self-hosted, VPC, air-gapped |
| LiteLLM | Python, Uvicorn | Yes | Yes | Latency-based | Keys, teams, orgs | Self-hosted, Cloud |
| OpenRouter | Hosted | Yes | Yes | Model-level | Keys, provider filters | Hosted |
| Vercel AI Gateway | Hosted | Yes | Yes | No | Vercel billing | Hosted |
| Cloudflare AI Gateway | Hosted | Yes | Yes | No | Workers-tied | Hosted |
| Portkey | Hosted (self-host enterprise) | Yes | Yes | Conditional routing | Workspaces, keys | Hosted, Enterprise self-host |
Vendor numbers are measured on different infrastructure and are not directly comparable. The useful question is which runtime and deployment model matches the rest of the stack.
How to choose
- Self-hosted, high-RPS, VPC-only: Bifrost. The runtime cost is low enough to ignore and the governance surface is the same as backend AI traffic.
- Python-SDK agents and a smaller fleet: LiteLLM, with the option to put Bifrost behind it later.
- Prototype or small team that needs a long model list fast: OpenRouter, planning to migrate as volume grows.
- Vercel-first product team: Vercel AI Gateway, with awareness of exit costs.
- Cloudflare-first stack: Cloudflare AI Gateway, likewise.
- Hosted gateway with governance but no self-hosting need yet: Portkey.
The router you can run where your app runs, with the governance model you already use, is the one that sticks. For the wider gateway landscape see the top AI gateways survey and the gateway comparison.
Sources
- Bifrost source repository Maxim AI
- Bifrost docs: AI gateway overview Maxim AI
- Bifrost docs: retries and fallbacks Maxim AI
- Bifrost docs: t3.xlarge benchmark Maxim AI
- Bifrost docs: virtual keys and budgets Maxim AI
- Bifrost docs: semantic caching Maxim AI
- Maxim AI: Bifrost AI gateway Maxim AI
- LiteLLM AI Gateway documentation BerriAI
- LiteLLM production deployment guidance BerriAI
- LiteLLM routing documentation BerriAI
- OpenRouter documentation OpenRouter
- OpenRouter pricing OpenRouter
- Vercel AI Gateway documentation Vercel
- Vercel AI Gateway pricing Vercel
- Cloudflare AI Gateway documentation Cloudflare
- Cloudflare AI Gateway Unified Billing Cloudflare
- Portkey documentation Portkey
- Portkey conditional routing Portkey
Frequently asked questions
What is an LLM router?
An LLM router is the component that picks a provider and model for each request and handles failure. In practice it is almost always part of an AI gateway, since the routing decision has to share state with virtual keys, budgets, rate limits and audit. The six products here are all gateways with routing as a central feature.
How is auto-routing different from weighted routing?
Weighted routing distributes a fixed share of traffic to each provider (70/30, 60/30/10). Auto-routing or complexity routing picks a model per request based on prompt content, cost, quality or latency estimates. Most production deployments start with weighted plus retry and add auto-routing later for cost reasons.
What is the best self-hosted LLM router?
Bifrost. It is Apache 2.0, Go, exposes one OpenAI-compatible API across 25+ providers and more than 10,000 models, and adds about 11 µs per request at 5,000 RPS in its published t3.xlarge benchmark. Weighted routing, retries and fallbacks are built in and share state with virtual keys and budgets.
Does an LLM router save money?
It can, but only with a policy. Saving comes from three places: semantic caching on near-duplicate prompts, fallback to a cheaper model when the primary is saturated or down, and complexity-aware routing that sends short prompts to small models. A router without one of these is just a load balancer.
Does an LLM router add latency?
It adds some. The key question is how much at the load you run. Bifrost's published figure is about 11 µs at 5,000 RPS; hosted routers add a network hop to a third-party edge. On the short tail of chat responses this is noise; on long completions it is invisible.
Can I switch routers without changing application code?
Usually yes, because most routers expose an OpenAI-compatible API. Changing `base_url` and the key moves traffic from one router to another. The work is in the config: provider keys, routing rules, budgets and audit integrations.