ToolsComparison
Five auto-routing tools for LLM cost and latency, ranked for production use
Five auto-routing tools for LLM cost and latency compared in November 2026: how Bifrost, LiteLLM, Portkey, Not Diamond and OpenRouter pick models per request.

TL;DR
- Five auto-routing tools cover most teams in November 2026. The useful split is gateways with routing primitives versus standalone routers that pick for you.
- Bifrost ranks first for production: Apache 2.0, Go, complexity routing and weighted load balancing on the same request path as virtual keys and semantic caching, with about 11 µs of added overhead per request at 5,000 RPS.
- LiteLLM and Portkey route inside their own gateways; Not Diamond is a standalone classifier; OpenRouter’s auto-router picks among its hosted providers.
- Auto-routing is a cost-optimisation layer on top of weighted routing. Start with weighted primary/fallback, add complexity routing once the first bill-shock is behind you.
What auto-routing actually buys
Three economic facts shape the auto-routing decision in 2026.
- Model-price spread is wide. A small model costs a fraction of a frontier model per token, and most prompts in production are short and easy. Sending everything to the frontier default is the usual starting mistake.
- Latency matters per-call, not on average. Routing a chat-completion to a fast small model makes the UI feel fast; routing a batch job to a cheaper model makes the budget last. Different routes, same tool.
- Quality floors are per-feature. A customer-facing summarisation feature has a different quality floor than an internal classification job. The router has to know which feature made the call.
Auto-routing is the control that turns those facts into configuration instead of rewrites. It has to share state with budgets and audit so that spend attribution still works, and it has to run cheap enough that the routing decision is not itself a latency problem.
How these five were scored
| Criterion | What was assessed |
|---|---|
| Decision surface | Weighted, fallback, complexity-aware, conditional |
| Classifier | Built-in, pluggable, hosted |
| Where it runs | In the gateway, as a library, as a hosted service |
| Governance integration | Shares virtual keys, budgets, rate limits |
| Caching | Semantic caching composable with routing |
| Deployment | Self-hosted, hosted, hybrid |
| Observability | OpenTelemetry, Prometheus, trace content |
Where the decision runs is weighted heaviest. A classifier that sits outside the gateway makes it harder to apply budgets, caches and audit consistently.
1. Bifrost — complexity routing on a fast request path
Bifrost’s routing primitives sit inside the same Go binary that does governance and observability. The retries and fallbacks page covers weighted routing across keys and providers with bounded retry chains; complexity routing lets teams pick a model per request based on prompt content and declared feature.
The useful detail is that the routing decision shares state with virtual keys and budgets. A route that downgrades from a frontier model to a small one on a soft budget cap is one config change away; a route that avoids a model entirely for a given customer is a virtual-key allow-list.
Runtime cost is low enough to design around. The t3.xlarge benchmark reports about 11 µs of added overhead per request at 5,000 RPS, with weighted key selection at about 10 ns. Peak memory is roughly 3.3 GB of the instance’s 16 GB. Semantic caching sits in front of the router: cache hits bypass the classifier, cache misses go through routing.
For a production deployment where routing has to coexist with budgets, audit and MCP governance, Bifrost is the only one of the five that treats all of them as one request path. Teams can start with weighted routing on day one and switch to complexity routing once the per-prompt classifier is tuned. The source repository is the starting point.
2. LiteLLM — routing primitives inside the Python proxy
LiteLLM’s routing page covers simple shuffle, weighted round-robin, latency-based routing and fallbacks. The proxy reliability page covers retries, cooldowns and health checks. For teams who already run LiteLLM, routing is a config change rather than a new piece of infrastructure.
Latency-based routing is the closest LiteLLM primitive to complexity routing; it picks the provider with the best observed latency, which is a decent proxy for load but not for prompt difficulty. Teams who need complexity-aware routing often add a classifier outside LiteLLM or switch to a gateway that bakes it in.
LiteLLM is a strong pick for Python-first teams who want the router as part of the same process their application already imports. The pod-per-worker scaling model remains the ceiling.
3. Portkey — conditional routing in a hosted gateway
Portkey’s conditional routing is rule-based: match on request headers, metadata or body, pick a provider or a model. Fallback and load balancing handle the usual retry and weighted-share cases.
The hosted model is the trade-off. Portkey sees the prompts and the completions, the routing decision runs in Portkey’s edge, and budgets and audit live there too. For teams who want hosted governance with real routing primitives, Portkey is the common pick; teams that need self-hosting switch to Bifrost. The Portkey alternatives survey covers the swap pattern.
4. Not Diamond — the standalone classifier
Not Diamond is a classifier that picks a model given a prompt and a list of candidates. It is not a gateway; it is the routing decision inside whatever gateway or application is making the call. Teams usually wire Not Diamond into a gateway as the step that chooses model before the gateway handles keys, budgets and audit.
The appeal is that Not Diamond specialises in the classifier, which is the hard part of auto-routing. The limitation is operational: Not Diamond is a hosted service, so the routing decision passes through a third party. Teams running air-gapped or sovereign-cloud workloads reach for an in-gateway option instead.
5. OpenRouter — auto-routing among hosted providers
OpenRouter’s model routing covers primary/fallback across its provider list; the auto-router picks a model per request based on OpenRouter’s internal classifier. For a prototype or small production app, it is the easiest way to get some auto-routing in exchange for OpenRouter’s fees.
The trade-offs are the usual hosted-aggregator ones: 5.5 per cent on Standard credit top-ups, 8 per cent on Business, 5 per cent on crypto, 5 per cent BYOK above $25,000/month of list-price inference. Production teams typically outgrow the auto-router around the same time they outgrow OpenRouter; the OpenRouter alternatives survey covers the migration.
Side-by-side
| Tool | Where it runs | Decision surface | Classifier | Governance | Deployment |
|---|---|---|---|---|---|
| Bifrost | In the gateway (Go, in-process) | Weighted, fallback, complexity | Built-in, pluggable | Shares virtual keys | Self-hosted, VPC, air-gapped |
| LiteLLM | In the proxy (Python) | Weighted, latency-based, fallback | Pluggable | Shares LiteLLM keys | Self-hosted, Cloud |
| Portkey | Hosted gateway | Weighted, conditional, fallback | Rule-based | Hosted workspaces | Hosted, Enterprise self-host |
| Not Diamond | Hosted classifier | Classifier only | Hosted ML | Outside the tool | Hosted |
| OpenRouter | Hosted aggregator | Auto-router, weighted, fallback | Hosted | Hosted key filters | Hosted |
Vendor numbers are measured on different infrastructure and are not directly comparable. The useful question is which decision surface fits the workload.
How to choose
- Production AI with budgets, audit and MCP already on a self-hosted gateway: Bifrost. The routing primitives share state with the governance primitives, which is where most auto-routing projects stall.
- Python-first teams with existing LiteLLM usage: LiteLLM, with a classifier plugged in for the complexity step. Switch when pod count hurts.
- Teams that want hosted governance with real routing rules: Portkey.
- Teams that want to specialise on the classifier: Not Diamond as the decision step inside a gateway that handles everything else.
- Prototypes and small production apps: OpenRouter’s auto-router, with a plan to migrate once volume grows.
Auto-routing is a layer on top of the boring stuff. Get weighted plus retries plus semantic caching working first; add complexity routing when there is a bill to beat.
Sources
- Bifrost docs: retries and fallbacks Maxim AI
- Bifrost docs: AI gateway overview Maxim AI
- Bifrost docs: t3.xlarge benchmark Maxim AI
- Bifrost docs: semantic caching Maxim AI
- Bifrost docs: virtual keys Maxim AI
- Maxim AI: Bifrost AI gateway Maxim AI
- Bifrost source repository Maxim AI
- LiteLLM routing BerriAI
- LiteLLM proxy routing BerriAI
- Portkey conditional routing Portkey
- Portkey fallback and load balancing Portkey
- Not Diamond documentation Not Diamond
- Not Diamond router Not Diamond
- OpenRouter routing OpenRouter
- OpenRouter auto-router OpenRouter
Frequently asked questions
What is LLM auto-routing?
Auto-routing is picking a model for each request automatically, usually to trade quality for cost or latency. A classifier scores the prompt, the router picks a small model for short/easy prompts and a bigger model for hard ones. The alternative is static weighted routing across providers.
Does auto-routing save money?
It can, but only with discipline. The saving comes from sending a majority of short prompts to a small cheap model and reserving the large model for the ones that need it. A badly tuned classifier that sends hard prompts to small models costs more in retries than it saves on cheap tokens.
What is the best self-hosted auto-routing tool?
Bifrost. Its complexity routing and weighted load balancing run inside the same Go gateway that handles governance and observability, so the routing decision shares state with virtual keys, budgets and audit. The published t3.xlarge benchmark reports about 11 µs of added overhead per request at 5,000 RPS.
Is Not Diamond a gateway or a classifier?
Not Diamond is a classifier that picks a model. Teams use it as a routing decision inside a gateway or in-app; it does not replace an AI gateway. For production use it is usually sat behind a gateway that handles keys, budgets and audit.
Why not use OpenRouter's auto-router?
OpenRouter's auto-router picks among its hosted providers, which is fine for prototypes. For production it has the usual hosted-aggregator trade-offs: third party in the data path, fees on credits and BYOK, no self-hosted option.
Can I combine semantic caching with auto-routing?
Yes, and most teams should. The gateway serves the cache hit before the router ever runs; cache misses go through the classifier. Bifrost's semantic cache uses exact-match and similarity search and composes naturally with its routing primitives.