Top 5 AI gateways for routing between vLLM, Ollama and cloud LLMs in 2026

An LLM router for hybrid inference is the layer that decides, request by request, whether a prompt runs on a self-hosted open-weight model or on a cloud API, and it is usually built into an AI gateway. The pattern has become common in 2026: teams serve open-weight models on their own GPUs with vLLM or SGLang for cost and data control, run Ollama on developer laptops and edge boxes, and keep cloud APIs for frontier-quality answers and traffic spikes. All three runtimes speak the OpenAI-compatible API, so the hard part is no longer the request format. It is routing by model, cost, load and health, and falling back cleanly from local to cloud when a GPU pool is full.

This ranking covers five gateways whose documentation describes routing to self-hosted inference servers alongside hosted providers. It is a spoke of the site’s broader list of open-source AI gateways you can self-host, narrowed to one question: which gateway handles a mixed fleet of vLLM, Ollama and cloud models best. Every capability below comes from the vendor’s own documentation, read between 8 and 10 October 2026.

How the five were ranked

The ranking uses seven criteria, each read from primary documentation. A gateway scores well when the capability is documented precisely enough to configure a local-plus-cloud fleet without guesswork.

Local runtime support and routing logic carry the most weight because they decide whether a self-hosted model is a first-class target or a generic URL with no runtime-specific handling. Readers new to the category may want the site’s explainer on what an AI gateway is first.

The five gateways at a glance

Figure 1: In a hybrid fleet the application keeps one request shape; the gateway decides whether it runs on a laptop, a GPU pool or a cloud API.

1. Bifrost

Bifrost is an open-source AI gateway written in Go and released under Apache 2.0 by Maxim AI, with a paid Enterprise tier. It exposes one OpenAI-compatible API across 25+ providers and 10,000+ models, and existing OpenAI, Anthropic and Google GenAI SDKs work by changing the base URL. For this ranking the relevant fact is that self-hosted runtimes are first-class providers, not a workaround. Bifrost gets the longest write-up below because its documentation covers every criterion in the table, including the local-runtime details the other four leave to generic configuration.

Local runtimes as named providers. The vLLM provider supports chat and text completions, the native Responses API (sent straight to /v1/responses rather than translated), embeddings, rerank, transcription and token counting. Each key carries its own server URL and model name, so two vLLM servers on different nodes become two weighted keys of one provider. The API key can be left empty for local servers, and Bifrost then omits the auth header. An optional use_anthropic_endpoints switch routes chat and Responses traffic through vLLM’s Anthropic-compatible /v1/messages endpoint, per key or per model alias. When a client such as a coding agent sends Anthropic server tools (web search, code execution and similar) that a self-hosted model cannot run, Bifrost drops them instead of forwarding a request the server would reject. That detail matters in practice: clients that enable web search by default would otherwise fail every call to a local model.

The Ollama provider covers chat, Responses, text completions, embeddings and model listing, filters parameters Ollama does not accept (such as store and service_tier), and supports tool calling. The SGLang provider offers the same OpenAI-compatible surface with the optional Anthropic-compatible mode and token counting. Anything else that speaks the OpenAI format, such as a llama.cpp server or a second vLLM fleet with different limits, can be added as a custom provider with its own name, base URL and list of allowed request types.

Routing that can see cost and load. Bifrost has two routing layers. Virtual keys carry static, weighted provider lists with fallbacks, which suits a fixed “70 per cent local, 30 per cent cloud” split. On top of that, routing rules evaluate CEL expressions before provider selection. A rule can read the requested model, request headers, the virtual key’s team or customer, and capacity metrics expressed as percentages: budget_used, tokens_used and request. The documentation’s own example sends traffic to a cheaper target when budget_used > 90. For hybrid fleets the same mechanism works in reverse: send everything to the local vLLM pool by default and spill to a cloud model when a header marks the request as long-context or premium, or when the local pool’s token rate limit passes a threshold. Rules are scoped (virtual key, team, customer, global), evaluated first-match-wins, and can be chained. A complexity_tier variable classifies each request as simple, medium or complex by embedding similarity, so easy prompts can stay on a small local model while hard ones go to a frontier API.

Retries and fallbacks that read the error. The retries and fallbacks page separates per-key failures (401, 402, 403, 429) from transient server failures (5xx, network, DNS). Transient failures retry the same key with exponential backoff (500 ms initial, 5 s cap by default); 429s rotate to another key but still back off; dead credentials rotate immediately. When retries are exhausted, the request moves to the next provider/model in the fallback chain, and every plugin (governance, caching, logging) runs again for the fallback. A chain of vllm/llama then sgl/llama then a cloud model is a single field in the request or the virtual key.

Overhead and operations. Bifrost’s published benchmark reports 11 µs of added overhead per request at 5,000 RPS on an AWS t3.xlarge. Per-provider concurrency and queue size are configurable, which helps when a local server can only take a few parallel requests. Prometheus metrics and OpenTelemetry export are both documented, and teams that want request-level tracing can pair the gateway with an AI observability stack.

Enterprise tier. Adaptive load balancing (weights adjusted from live error rates and latency), the header-driven circuit breaker, clustering, SSO, RBAC, audit logs, guardrails and in-VPC deployment come with Bifrost Enterprise.

Best for: teams running a mixed fleet of vLLM, SGLang, Ollama and cloud APIs that want local servers treated as first-class providers, with routing rules that can react to budget and load.

2. LiteLLM

LiteLLM is a Python SDK and proxy from BerriAI that exposes an OpenAI-compatible interface over a long list of providers. It is the most common choice when a Python team first puts Ollama and a cloud key behind one endpoint, and its YAML model list is easy to read.

Strengths. Local runtimes are well covered. The vLLM provider page uses the hosted_vllm/ prefix plus an api_base for vLLM’s OpenAI-compatible server (the older vllm/ in-process route is marked deprecated). The Ollama provider supports ollama/ and ollama_chat/, and the docs recommend ollama_chat for better responses. Any other OpenAI-compatible server can be reached with the openai/ prefix and a custom base URL. The router offers weighted pick (the default, recommended for production), rate-limit-aware, latency-based, least-busy, lowest-cost and custom strategies, so a local deployment and a cloud deployment of “the same” model can share a model group. Fallbacks move from one model group to another after num_retries, and Redis can track cooldowns and usage across proxy instances. The health endpoint sends a real test request to each configured model, and background health checks can run every 300 seconds by default and serve cached results.

Trade-offs. LiteLLM’s health checks are model probes that cost a few tokens per model, rather than lightweight liveness checks against an inference server, so teams with many local deployments tune the interval or exclude models. The documentation itself warns that usage-based routing is not recommended for production because of its performance cost. Running several proxies with shared cooldowns and limits means operating Redis as well. Supply-chain hygiene also matters: in March 2026, PyPI releases 1.82.7 and 1.82.8 carried malicious code for about 40 minutes before quarantine, although the official proxy Docker image was not affected and a clean 1.83.0 followed on 30 March.

Best for: Python-first teams that want the broadest provider list and a quick way to put Ollama and vLLM next to cloud keys, and that are comfortable running Redis and pinning releases. The site’s LiteLLM alternatives guide covers what teams move to when that operating bar is too high.

3. Kong AI Gateway

Kong AI Gateway is a set of AI capabilities on Kong Gateway, configured through AI Model Provider and AI Model entities or through the AI Proxy plugins. Its provider list now includes named vLLM and Ollama providers alongside OpenAI, Anthropic, Bedrock, Gemini, Mistral, Hugging Face and others. For a self-hosted server, the documented step is to set upstream_url on the AI Model to the server’s address.

Strengths. Load balancing is the most varied of the five. Kong documents seven algorithms: weighted round-robin, consistent hashing, least-connections, lowest-usage, lowest-latency (measured by default as time per output token), semantic and priority. Priority groups fit hybrid inference neatly: put the local vLLM targets in the top group and the cloud model in a lower group, and the cloud only receives traffic when every local target is unavailable. Failover is opt-in per error class through failover_criteria (for example http_429 and http_500), and a circuit breaker marks a target unhealthy after max_fails failures for fail_timeout seconds. Teams already on Kong get the same consumer, plugin and observability model for AI traffic as for the rest of their APIs, with Konnect available as a managed control plane.

Trade-offs. The free AI Proxy plugin lists vLLM and Ollama among its providers, but the multi-target plugin, AI Proxy Advanced, is “only available as part of our AI Gateway Enterprise offering”, according to its plugin page, so teams should budget for the paid tier before designing a local-first, cloud-second setup on Kong. If every target fails at once, the documentation says requests return HTTP 500. Kong publishes no AI-specific overhead figure.

Best for: organisations already running Kong Enterprise or Konnect that want local and cloud models governed by the same gateway, consumers and plugins as the rest of their API estate.

4. Apache APISIX AI gateway

Apache APISIX handles model traffic through its ai-proxy and ai-proxy-multi plugins. The multi-instance plugin lists OpenAI, DeepSeek, Azure OpenAI, AIMLAPI, Anthropic, OpenRouter, Gemini, Vertex AI and Bedrock, plus an openai-compatible provider whose override.endpoint points at any compatible server. That is how vLLM, SGLang and Ollama are reached: as OpenAI-compatible instances with their own URLs.

Strengths. Health handling is a real strength for local servers. Each instance can carry active health checks (HTTP, HTTPS or TCP), and the docs note that while OpenAI and DeepSeek publish no health endpoint, OpenAI-compatible servers often do; vLLM’s server exposes a /health endpoint. Instances carry a priority that takes precedence over weight, so a local pool can be primary and a cloud instance secondary. fallback_strategy can trigger on exhausted rate-limit budgets, HTTP 429 and 5xx, with extra status codes added through fallback_http_statuses, a max_retries cap and a retry_on_failure_within_ms setting that retries fast failures but not slow ones. Balancing algorithms include weighted round-robin, consistent hashing and semantic selection. APISIX is an Apache Software Foundation project released under Apache 2.0.

Trade-offs. Local runtimes are generic OpenAI-compatible instances rather than named providers, so runtime-specific behaviour (vLLM’s Anthropic-compatible endpoint, Ollama’s parameter quirks) is left to the operator. There is no hierarchy of virtual keys, teams and budgets in the plugin itself; consumer and rate-limit plugins fill that role. APISIX publishes no AI-specific overhead figure.

Best for: teams already operating APISIX that want health-checked local instances with priority-based cloud fallback without buying a commercial tier.

5. Agent Router (formerly Envoy AI Gateway)

Agent Router is the project created as Envoy AI Gateway, built as a control plane on Envoy Gateway for Kubernetes. Its documentation lists “resilient connectivity across providers and self-hosted models” as a core goal, and its supported-providers table includes a self-hosted-models entry that uses the OpenAI schema, noting that vLLM speaks that format and that API-key auth can be added. A local quick start (aigw run) starts an OpenAI-compatible router on port 1975 for testing.

Strengths. Agent Router’s distinguishing feature for self-hosted inference is InferencePool support. It integrates with the Gateway API Inference Extension, in which an Endpoint Picker reads metrics from the model servers in a pool and tells the gateway which replica should take each request. That puts model-aware, load-aware replica selection and provider failover in one Kubernetes-native stack. The project’s stated goals include automatic failover, upstream authentication to external providers, rate limiting and a policy framework for usage limits, and developers see one OpenAI-compatible API.

Trade-offs. It is built for Kubernetes and Envoy; teams without that platform take on both. Self-hosted support is generic (the docs mark it with a warning that behaviour depends on the API schema the server speaks), and there are no named Ollama or SGLang providers. Budgets are expressed as rate-limit and policy resources rather than a key, team and customer hierarchy, and no overhead figure is published.

Best for: platform teams running Envoy Gateway on Kubernetes that want replica-level inference scheduling and cloud provider routing declared in the same stack.

What actually differs between them

The five gateways agree on the request format and disagree on almost everything else. The differences that matter for a vLLM, Ollama and cloud fleet are how each gateway models local runtimes, what makes it move to another backend, and where it sits relative to in-cluster routers.

vLLM vs Ollama as gateway backends

vLLM and Ollama solve different problems, and the gateway configuration should reflect that. vLLM’s OpenAI-compatible server exposes completions, chat, the Responses API, embeddings, transcription, translation, rerank and an Anthropic-compatible messages endpoint, and it supports data-parallel deployment across GPUs. It is built for many concurrent requests on data-centre hardware.

Ollama is built for one machine. Its OpenAI compatibility layer serves /v1/chat/completions on localhost:11434, and the client needs an API key value that Ollama ignores. The Ollama FAQ is explicit about limits: OLLAMA_NUM_PARALLEL defaults to one request per model at a time, OLLAMA_MAX_QUEUE defaults to 512 queued requests, and once the queue is full the server answers with HTTP 503. A new model that does not fit in VRAM waits until another unloads.

Those numbers shape gateway rules. An Ollama box on a developer laptop or an edge site should be a target with a short timeout and a fallback, so a 503 or a slow model load becomes a cloud call instead of a stuck request. A vLLM pool can take a larger share of steady traffic, with fallback reserved for real outages or overload. SGLang sits closer to vLLM: a high-throughput server with OpenAI-compatible completions, vision and embeddings endpoints and an Anthropic-compatible messages endpoint.

Named providers vs generic OpenAI-compatible endpoints

Every gateway here can reach a local server, but only some model it as a distinct runtime. Bifrost, Kong and LiteLLM have named vLLM and Ollama routes; Bifrost also names SGLang. APISIX and Agent Router treat local servers as OpenAI-compatible endpoints with a custom URL.

The difference shows up at the edges. A named provider can strip parameters the server rejects, drop Anthropic server tools that a self-hosted model cannot execute, or switch to the server’s Anthropic-compatible endpoint for clients that send Messages API calls. With a generic endpoint, those adjustments become the operator’s job, usually discovered when a coding agent’s first request fails.

What triggers a move to another backend

Fallback behaviour is where hybrid routing succeeds or fails. The useful questions are which signals each gateway reads and how quickly it gives up on a local target.

Figure 2: A local-first chain only works if the gateway can see the status code or timeout that should move the request to the next tier.

Two details are easy to miss. First, a fallback is a new request to a different model, so prompt templates, tool schemas and context limits must work on both. A 128k-token prompt that fits a cloud model will not fit a local model served with a short context, and falling back in that direction fails for a different reason. Second, cost attribution has to follow the request. When a fallback moves a call from a free local GPU to a paid API, the log line should record the final provider and its price, or the cost of fallbacks stays invisible until the invoice arrives. For a deeper look at how gateways classify errors, the site’s five-way LLM gateway comparison tests failover semantics in detail.

The vLLM router and the AI gateway are different layers

The term “vLLM router” now covers two things, and confusing them leads to awkward architectures. The vLLM Router from the vLLM project is a lightweight forwarder for large vLLM deployments, with cache-aware, power-of-two, consistent-hashing, random and round-robin balancing and support for prefill/decode disaggregation. The vLLM production stack and the Kubernetes Gateway API Inference Extension play a similar role: they pick the best replica inside one pool of servers running one model, using metrics such as queue depth and KV-cache state.

An AI gateway sits one level above. It decides which pool, provider or cloud model should handle a request at all, and applies keys, budgets and logging on the way. In a mature hybrid setup both layers exist: the gateway sends “llama-70b” traffic to the in-cluster router, which picks a replica, and the gateway falls back to a cloud model if the whole pool returns errors. Agent Router is the one entry that ships both roles in one stack through InferencePool; the others call an in-cluster router as just another OpenAI-compatible target.

Governance applies to local traffic too

Self-hosted inference is not free. GPUs have a cost per hour, and a local pool has a fixed capacity that one noisy team can exhaust. Gateways that apply the same virtual keys, budgets and rate limits to local and cloud targets let platform teams give each team a share of local capacity and a cloud budget for overflow. Bifrost’s routing rules read those limits directly (tokens_used, request, budget_used), LiteLLM tracks TPM and RPM per deployment, and Kong and APISIX rely on their rate-limiting plugins. For background on why gateways are a natural place for this, a production-ready comparison of open-source LLM gateways for self-hosted deployments covers the operational side of running one inside a private network.

Recommendations by constraint

The right gateway depends more on what a team already runs than on any single feature. The following pairings follow from the criteria and the documentation above.

Figure 3: The existing platform usually decides the gateway; a dedicated gateway is the default when there is no platform to extend.

• Starting fresh with a mixed fleet. Bifrost is the strongest default: named vLLM, SGLang and Ollama providers, CEL routing rules that can read budget and rate-limit usage, and error-class-aware retries and fallback chains.
• Python team, prototype to small production. LiteLLM puts hosted_vllm/ and ollama_chat/ next to cloud keys in a few lines of YAML; plan for Redis, pinned versions and tuned health-check intervals as traffic grows.
• Kong already in production. Use Kong’s vLLM and Ollama providers with priority-group balancing so cloud models only serve traffic when local targets fail, budgeting for AI Gateway Enterprise.
• APISIX already in production. ai-proxy-multi with active health checks on local instances and priority fallback to a cloud instance covers the core pattern with no paid tier.
• Kubernetes platform with large vLLM pools. Agent Router with InferencePool puts replica scheduling and provider failover in one declarative stack; add a dedicated gateway in front if application teams need key-level budgets.
• Developer laptops and edge boxes. Point local tools at the gateway rather than at localhost:11434, give the Ollama target a short timeout, and fall back to a shared vLLM pool or a cloud model so a busy laptop never blocks work.

Whatever the choice, test the fallback path before it is needed. Stop the local server, saturate an Ollama queue until it returns 503, and confirm that requests reach the cloud target with the right model, prompt format and cost record.

Verdict

Hybrid inference has moved from experiment to default architecture for teams that serve open-weight models, and the LLM router in front of that fleet now matters as much as the inference servers behind it. All five gateways can reach vLLM, SGLang and Ollama through the OpenAI-compatible API; they differ in how much runtime-specific handling they apply to those servers and how well they move traffic when a local tier fails.

We rank Bifrost first because it treats local runtimes as named providers with runtime-specific handling, lets routing rules react to budget, load and request complexity, and documents exactly which errors cause a retry, a key rotation or a fallback. LiteLLM is the quickest path for Python teams, Kong and APISIX are the right extensions for organisations already running those gateways, and Agent Router is the natural fit when replica scheduling inside Kubernetes is part of the problem. The deciding question is simple: which layer will own the decision to leave the local GPU and pay for a cloud token, and can that layer see the signal that should trigger it.