Five LLM gateways compared: failover, governance, MCP and where each one fits

Once a company is calling more than one model provider from more than one application, the same code appears everywhere: credential handling, retries, fallbacks, spend tracking, logging. An LLM gateway collapses that into one service that every application talks to. The category has matured quickly, and by September 2026 the products differ far less on the basic feature list than on how they classify errors, where they run, what they charge and what they leave to the enterprise tier.

This comparison looks at five gateways that between them cover the main architectural choices: two open-source, self-hostable gateways (Bifrost and LiteLLM), one extension of an existing API gateway (Kong AI Gateway), one edge-network service (Cloudflare AI Gateway) and one hosted control plane aimed at application developers (Vercel AI Gateway). Every product claim below comes from the vendor’s own documentation, read in September 2026, and every benchmark figure is labelled as the vendor’s own.

The short version: Bifrost is the strongest option where throughput, error semantics and governance depth matter and you are willing to run a binary. LiteLLM is the widest and most familiar to Python teams. Kong is the right answer for organisations already standardised on Kong, with the caveat that most AI features sit behind an enterprise licence. Cloudflare and Vercel remove operations entirely, at the cost of a network hop and a much thinner policy model.

Figure 1: The value of a gateway is that this path is configured once instead of reimplemented in every application.

What a gateway does, and the one question that decides the choice

A gateway is a reverse proxy that understands model APIs. The application sends an OpenAI-style, Anthropic-style or gateway-native request to one endpoint with one credential; the gateway resolves who is calling, checks that caller’s budget and rate limit, picks a provider key, calls the provider, handles the error if there is one, and records tokens, cost and latency. Everything in that sentence is something every application would otherwise implement on its own.

The reason this matters more than it did two years ago is that the providers have made rate limiting a first-class concern. OpenAI enforces limits at the organisation and project level in requests per minute, tokens per minute, requests per day, tokens per day and images per minute, graduates accounts through usage tiers by cumulative spend, and tells clients to honour Retry-After and back off exponentially with jitter. Anthropic enforces requests, input tokens and output tokens per minute using a token-bucket algorithm, returns a retry-after header on a 429, and adds “acceleration limits” that trigger when an organisation’s usage ramps sharply. Anthropic’s monthly spend cap also returns a 429, but without a retry-after header, so a naive retry loop spins until the first of the month. A gateway that treats every 429 identically will get this wrong. That is the mechanism behind most of what follows.

The one-line criterion

Choose the gateway whose deployment model you can accept, then pick among the survivors on error semantics and governance. Feature lists converge; the deployment constraint and the failure path do not. Figure 3 near the end lays this out as a decision.

The contenders

The fifth seat could have gone to OpenRouter or to Azure API Management’s AI gateway capabilities. We chose Vercel because its documentation is the clearest of the three on failover order, budget semantics and data handling, which are the things this comparison is about. OpenRouter appears below only as one of the providers Bifrost can route to.

Request path and overhead at load

Every gateway adds latency. The question is how much, how it scales with requests per second, and whether the published number was measured in conditions like yours.

Bifrost’s benchmark numbers

Bifrost is open-source, written in Go by Maxim AI, and installs with npx -y @maximhq/bifrost or docker run -p 8080:8080 maximhq/bifrost. The web UI and the API share port 8080, configuration lives in config.json, and the unified endpoint is POST /v1/chat/completions with models named provider/model, for example openai/gpt-4o-mini. Existing OpenAI, Anthropic and Google GenAI SDK code moves over by changing base_url to a per-provider path such as http://localhost:8080/openai; there are also LiteLLM, LangChain and PydanticAI integrations. The docs list two dozen providers, including OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, Azure OpenAI, Gemini, Groq, Mistral, Cohere, Cerebras, Databricks, DeepSeek, Ollama, Hugging Face, OpenRouter, Perplexity, xAI, Replicate, SGLang and vLLM, and the product page claims “1000+ models” and “1,000+ teams”, both vendor figures.

Bifrost’s published benchmarks report, at 5,000 requests per second on an AWS t3.xlarge (4 vCPU, 16 GB), 11 µs of gateway overhead, a 100 per cent success rate, 1.67 µs average queue wait and 3,340 MB peak memory. On a t3.medium (2 vCPU, 4 GB) at the same rate the overhead is 59 µs and peak memory 1,313 MB. The benchmarking page is explicit that these runs used mocked OpenAI calls, and notes that the t3.xlarge run used roughly 10 KB response payloads against roughly 1 KB on the t3.medium.

The same page publishes a head-to-head against LiteLLM at 500 requests per second on a t3.medium in us-east-1, over 60 seconds with 500 concurrent virtual users and real Tier 5 OpenAI access: P50 latency 804 ms versus 38.65 s, P99 1.68 s versus 90.72 s, throughput 424 versus 44.84 requests per second, peak memory 120 MB versus 372 MB, and success rate 100 per cent versus 88.78 per cent. Gateway overhead measured against a 60 ms mock response was 0.99 ms versus 40 ms. On identical hardware and load, that is roughly a 54-fold difference at the tail and more than nine times the throughput. The harness is public, and the benchmarking guide invites teams to validate higher targets on their own instance sizes, provider mix and payload profile.

LiteLLM: what it takes to run at scale

The production guidance explains where the resources go: a floor of 1 vCPU and 4 GiB per pod, one Uvicorn worker per pod on Kubernetes, PostgreSQL for keys and spend logs, Redis 7.0 or newer “as soon as you run more than one proxy instance” for rate limits and router state, and a Redis transaction buffer for spend writes at roughly 1,000 requests per second or ten-plus instances. None of that is wrong, but it is a different cost of ownership from a single Go binary, and it is the deployment the Bifrost head-to-head ran against: one Python instance doing work that Bifrost handled with a fraction of the memory.

Kong, Cloudflare and Vercel

Kong publishes no AI-specific throughput benchmark that we could find in its AI Gateway documentation; the AI plugins run inside the Kong Gateway data plane, so the overhead is whatever Kong adds plus the plugin work, and it inherits Kong’s scaling model. Cloudflare and Vercel do not publish gateway overhead figures either. For a hosted gateway the relevant number is the network round trip from your application to the gateway’s nearest point of presence, which depends on where you run and which neither vendor can state in advance. Vercel’s documentation notes that a request “can terminate and be processed in any Vercel region before AI Gateway forwards it to the provider” and that region-pinned gateway hosts are “coming”.

If you serve a chat product at low volume, none of this matters. If you serve an agent platform that makes millions of tool-calling requests a day, the gateway sits inside every one of those loops, and the difference between microseconds and tens of milliseconds compounds. Agent frameworks make the number of model calls per user action a design choice, and the gateway pays for it.

Failover: what each gateway does with a 429, a 401 and a 500

All five retry and fall back. The differences are in what they distinguish, and this is where reading the documentation pays off most.

Bifrost: error classes decide the action

Bifrost’s fallback semantics are the most explicit of the five. A request carries a fallbacks array of provider/model strings. Network failures and 5xx responses are retried against the same provider with exponential backoff governed by network_config.max_retries (default 0), retry_backoff_initial (default 500 ms) and retry_backoff_max (default 5,000 ms). A 429 rotates to the next key within the provider and backs off. A 401, 403 or 402 rotates to the next key without backing off, since waiting will not revive a revoked or unfunded credential. A 400, 404 or 422 is treated as a client error: no retry, straight to the next fallback. Each provider in the chain gets its own full retry budget, so a three-entry chain with max_retries: 2 can make up to nine attempts. The response’s extra_fields.provider reports who actually answered, and a plugin can set AllowFallbacks = false on an error to stop the chain, which is how a compliance block avoids being “fixed” by falling to a less restricted provider.

{
  "model": "openai/gpt-4o-mini",
  "messages": [{ "role": "user", "content": "Summarise this ticket." }],
  "fallbacks": [
    "anthropic/claude-3-5-sonnet-20241022",
    "bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
  ]
}

LiteLLM: cooldowns and typed fallback lists

LiteLLM’s router works in terms of deployments: several entries can share a model_name, and the router picks among them. On failure it retries num_retries times (settable per request via x-litellm-num-retries, per deployment or globally), and a deployment that fails more than allowed_fails times in a minute enters a cooldown for cooldown_time seconds. The docs classify 429 and 5xx as retryable and 401, 404, 408, context-window and content-policy errors as non-retryable; the non-retryable classes cool the deployment down immediately. A RetryPolicy can set different retry counts per error type. Fallbacks are typed: fallbacks for general errors, context_window_fallbacks and content_policy_fallbacks for those two specific cases, and default_fallbacks as a catch-all. disable_fallbacks: true can be set per request or per key, and enforce_fallback_model_access decides whether a key’s model allowlist applies to the fallback target. The distinction LiteLLM does not draw as cleanly as Bifrost is between “rotate key” and “retry”: key selection is a routing concern across deployments rather than a step inside one provider’s error path.

Kong: opt in to each status code

Kong’s AI Proxy Advanced plugin carries the balancer. Its failover_criteria defaults to ["error", "timeout"] and accepts http_403, http_404, http_429, http_500, http_502, http_503, http_504, invalid_header and non_idempotent; retries defaults to 5; and a circuit breaker (Kong 3.13 and later) marks a target unavailable after max_fails failures (default 0, which disables it) for fail_timeout milliseconds (default 10,000). The documentation states plainly that “client errors don’t trigger failover” unless you add the codes. Since 3.10 fallback works across providers with different request formats, so an OpenAI target can fail over to a Mistral one. The important operational fact is that the free ai-proxy plugin targets one model per route and returns a 400 if the request names a different model; failover and load balancing require the enterprise-tier advanced plugin.

Cloudflare and Vercel: ordered lists

Cloudflare’s Universal Endpoint takes an array of provider requests and falls to the next entry “if the first provider fails”; retries run first, with maxAttempts up to 5, retryDelay up to 60 seconds and constant, linear or exponential backoff, and the cf-aig-step header reports which entry answered. That endpoint is now marked deprecated in favour of the OpenAI-compatible endpoint plus Dynamic Routing, where a visual or JSON flow chains model nodes, conditional branches, percentage splits, rate-limit nodes and budget-limit nodes, and is invoked by using dynamic/route-name as the model. Neither page documents error classification; a failure is a failure.

Vercel’s gateway chooses providers dynamically “based on recent uptime and latency” unless you set order, only or sort (cost, ttft or tps) under providerOptions.gateway. Providers carry a health state: degraded and recovering providers are penalised in the sort, and down providers go last. Model fallbacks add a models array: the gateway tries every provider for the primary model, then every provider for the next model. Every attempt is recorded in providerMetadata.gateway.routing.modelAttempts, with status codes and response times per provider attempt, which is the best per-request failover trace of the five. What Vercel does not expose is a distinction between a 429 and a 500 on the way through; both simply advance the list.

Figure 2: The gateways agree on what a failure is and disagree on what to do about each kind.

Why the distinction matters

Recall the provider behaviour. Anthropic’s spend-cap 429 has no retry-after and will not clear until the month rolls over; OpenAI’s per-project 429 will clear in seconds. A gateway that rotates keys on 429 (Bifrost) survives a saturated key by moving to a sibling; a gateway that only cools down the deployment (LiteLLM) survives by moving to a sibling deployment, which amounts to the same thing if you have configured one per key; a gateway that treats 429 as generic failure (Cloudflare, Vercel) survives only if the next list entry is a different account. All three work; they demand different configuration discipline.

Load balancing, keys and governance

Governance means: who can call what, with what budget, and what gets logged. This is where the open-source gateways have pulled ahead of the hosted ones, and where Bifrost is deepest.

Keys and load balancing

Bifrost distributes requests across keys by weighted random selection: each key has a weight, the gateway sums the weights of eligible keys and draws. A key’s models array is an allowlist (since v1.5.0 an empty array denies everything and ["*"] allows all), blacklisted_models is a denylist that wins over the allowlist, and both accept regex: entries matched as RE2 patterns case-insensitively. A request can pin a key with x-bf-api-key (by name) or x-bf-api-key-id, bypassing selection. The enterprise tier adds adaptive load balancing, which scores providers and keys on error rate, latency against peers and history, and utilisation, and moves routes through Healthy, Degraded, Failed and Recovering states without operator action.

LiteLLM’s strategies are simple-shuffle (weighted by rpm or tpm, recommended), usage-based routing via Redis, latency-based, least-busy and cost-based, plus an order field for strict priority and enable_weighted_failover to retry within a model group before crossing groups. Kong’s advanced plugin offers round-robin, consistent-hashing on a header (default X-Kong-LLM-Request-ID), least-connections, lowest-latency using an exponentially weighted moving average of either time per output token or end-to-end latency, lowest-usage by tokens or cost, semantic routing against a Redis or pgvector store, and priority groups. Vercel’s sort by cost, time to first token or tokens per second is the hosted equivalent; Cloudflare’s percentage-split node covers A/B and gradual rollout.

Virtual keys, budgets and rate limits

Bifrost’s virtual keys sit in a three-level hierarchy: a key belongs to a team or a customer or neither, budgets attach at every level and cascade, and rate limits attach only at the key. Budgets are arrays, so one key can carry a daily and a monthly cap. A key can carry expires_at; an expired key is rejected with a 403 but kept for audit. Provider access is deny-by-default unless allow_all_providers is set, and the error taxonomy is precise: budget_exceeded is a 402, token_limited and request_limited are 429s, model_blocked and provider_blocked are 403s. Setting enforce_auth_on_inference: true makes a key mandatory on every inference call. The key can travel in x-bf-vk, Authorization: Bearer, x-api-key, x-goog-api-key or api-key, which is what lets unmodified OpenAI, Anthropic and Google SDKs authenticate against the gateway.

LiteLLM’s virtual keys are generated at /key/generate, require a PostgreSQL database and a master key, and carry max_budget, budget_duration, tpm_limit, rpm_limit and a models list; keys belong to users, teams or service accounts, and model access inherits through teams. Scheduled key rotation, IP allowlists and organisation-level admin roles are enterprise features.

Vercel’s budgets are the best-documented of the hosted pair: team, project, API key and user scopes stack, a request must pass every budget in scope, periods reset at UTC boundaries, defaults can cap every key or project that lacks a custom budget, and spend alerts fire at 50, 75 and 100 per cent. Two limits worth knowing: a budget is a soft cap because the check runs before the request and the crossing request completes, and BYOK spend is metered separately and “can’t be used to cap BYOK spend”. Cloudflare’s rate limit is one setting per gateway applied uniformly to every request; per-caller quotas exist only as nodes inside a Dynamic Route.

Guardrails

Bifrost’s enterprise guardrails cover 16 providers: Bifrost-managed secrets detection (Gitleaks-backed), custom regex and LLM-as-judge; Presidio, Azure AI Language PII and Azure Content Safety; and AWS Bedrock Guardrails, Google Model Armor, CrowdStrike AIDR, Gray Swan Cygnal, Patronus AI, Check Point, Repello Argus and Singulr AI. Rules are written in CEL and can target a team, a header or a provider; they run on input, output or both; and the actions are block (HTTP 446), redact (runtime, logs-only or reversible), detect-and-log (HTTP 246) or allow. A block prevents the model call or the MCP tool execution outright.

LiteLLM’s guardrails integrate Aporia, Lakera, Presidio, AWS Bedrock, Azure Content Safety, Guardrails AI, Cato Networks, OpenAI moderation and a generic API, in pre_call, post_call, during_call and logging_only modes; default-on guardrails and per-model assignment are open source, while per-key control, tag-based selection and policies are enterprise. Kong ships AWS Guardrails, Azure Content Safety, GCP Model Armor, Lakera Guard, a PII sanitiser and a custom-guardrail plugin, alongside the regex-based ai-prompt-guard (400 on a deny match) and the enterprise ai-semantic-prompt-guard (403 on semantic match against a vector store). Cloudflare’s Guardrails evaluate prompts and responses for text models and prompts only for embeddings, with flag, block or ignore per category, and are billed as Workers AI inference; they do not support stream: true. Vercel’s security pages cover zero data retention, prompt-training opt-out and provider and model allowlists rather than content guardrails.

Policy as code, and policy at the endpoint

Kong is the natural home of policy-as-code: every AI feature is a plugin in the same declarative configuration as the rest of the API estate, applied to routes, services, consumers and consumer groups. Bifrost configures through config.json, the web UI, the API or Helm, with guardrail rules in CEL; LiteLLM through config.yaml and the UI; Cloudflare and Vercel through dashboards, APIs and, in Vercel’s case, a CLI (vercel ai-gateway budgets set team --limit 500 --refresh-period monthly).

Bifrost also extends the same governance beyond the server. Bifrost Edge is an endpoint agent for macOS, Windows and Linux that routes traffic from desktop AI apps, browser AI, coding agents such as Cursor and MCP servers on employee machines through the gateway, so that virtual keys, budgets, audit logs and guardrails apply to AI use that never touched an application you wrote; it deploys through Jamf, Intune or Kandji, and the Edge overview has the current list of supported apps. It is the only one of the five that reaches the laptop.

Caching, MCP and tool routing

Caching

Bifrost’s semantic cache has two paths: a direct hash match on the normalised request, and an embedding-similarity match that needs an embedding-capable provider. It requires a vector store (Redis or Valkey, Weaviate, Qdrant or Pinecone), defaults to a five-minute TTL and a 0.8 similarity threshold, and skips conversations longer than three messages by default. Caching engages only when the request carries x-bf-cache-key (or a default_cache_key is configured); x-bf-cache-ttl and x-bf-cache-threshold override per request, x-bf-cache-type forces one path, and x-bf-cache-no-store reads without writing. Responses carry cache_debug with the hit type and similarity score. The plugin covers chat, Responses API, embeddings, transcription, speech and image generation, including streaming variants.

LiteLLM caches in memory, on disk, in Redis, Valkey, ElastiCache, Memorystore, S3 or GCS, with semantic variants on Redis, Valkey and Qdrant, and its docs note that an in-memory cache “lives inside one worker process”, so a four-worker proxy keeps four caches. Kong’s ai-semantic-cache (enterprise) does exact and semantic caching on Redis-compatible stores or pgvector and honours Cache-Control max-age and s-maxage. Cloudflare’s cache is exact-match only, off by default, with cf-aig-cache-ttl between 60 seconds and one month, cf-aig-skip-cache, a custom cf-aig-cache-key, and a cf-aig-cache-status header of HIT or MISS. Vercel’s documentation mentions caching among its model-and-provider settings but its prompt caching is the provider’s own, surfaced through cache_read and cache_creation token counts.

Anthropic’s rate-limit page is a reminder of why caching is a rate-limit tool as well as a cost tool: for most Claude models, cache-read input tokens do not count towards the input-tokens-per-minute limit, so a gateway that preserves prompt-cache breakpoints when it retries or falls over is worth more than its cache hit rate suggests.

MCP and tool routing

The Model Context Protocol is a JSON-RPC 2.0 protocol in which hosts run clients that connect to servers exposing tools, resources and prompts. Its specification says hosts “must obtain explicit user consent before invoking any tool” and that tool descriptions should be treated as untrusted unless the server is trusted. A gateway that terminates MCP is therefore a policy point, not just a proxy.

Bifrost’s MCP gateway is both an MCP client to upstream servers over STDIO, HTTP and SSE and an MCP server to downstream clients over POST /mcp (JSON-RPC) and GET /mcp (SSE), so Claude Desktop, Cursor or a custom application sees one aggregated tool set. Tool access is filtered per virtual key, and the x-bf-mcp-include-tools header narrows it further. Agent Mode, in which Bifrost runs the LLM loop itself, honours tools_to_auto_execute; in gateway mode that field is ignored because approval is the client’s job, which is the right reading of the spec. Code Mode, available from v1.4.0 in gateway deployments, lets the model write Python that orchestrates several tools in a sandbox; the docs claim “up to 92.8%” fewer input tokens versus conventional tool calling with three or more servers, a vendor figure we have not verified. Upstream servers are health-checked every 10 seconds with a 5-second timeout and marked unstable after five consecutive failures.

LiteLLM’s MCP gateway supports Streamable HTTP, SSE and stdio servers, controls access by key, team and organisation, forwards OAuth 2.0, bearer, basic and AWS SigV4 credentials, tracks cost per tool invocation, exposes /mcp-rest/tools/list and /mcp-rest/tools/call, and integrates with /chat/completions and /v1/responses; it is available in the open-source tier. Kong’s AI MCP Proxy (Kong 3.12 and later, enterprise) converts REST APIs into MCP tools, proxies third-party MCP servers over Streamable HTTP and aggregates tool sets behind consumer and consumer-group ACLs, with a companion AI MCP OAuth2 plugin. Cloudflare’s and Vercel’s AI Gateway documentation, as read for this piece, does not describe an MCP gateway.

Observability, deployment and data residency

Observability

Bifrost exposes GET /metrics in Prometheus format with bifrost_upstream_latency_seconds, bifrost_overhead_latency_microseconds, bifrost_input_tokens_total, bifrost_output_tokens_total, bifrost_cost_total, bifrost_cache_hits_total, bifrost_provider_key_up and bifrost_key_rotation_events_total, labelled by provider, model, virtual key, team and customer; the OpenTelemetry plugin exports OTLP over HTTP (port 4318) or gRPC (4317) using the gen_ai.* semantic conventions and attaches session.id from an x-bf-session-id header. Log exports to S3, GCS and BigQuery, a Datadog connector and audit logs are enterprise features. LiteLLM exposes /metrics in the open-source build with spend, token, latency, budget and cache metrics, and its otel callback exports OTLP over HTTP or gRPC and honours an incoming W3C traceparent. Kong’s Prometheus plugin, with config.ai_metrics set, emits kong_ai_llm_requests_total, kong_ai_llm_cost_total, kong_ai_llm_tokens_total and, from 3.8, kong_ai_llm_latency and cache latency metrics, though the plugin is not supported on Kong’s dedicated or serverless cloud gateways.

Cloudflare offers dashboard analytics, a GraphQL API and, on the Workers Paid plan, Logpush; logs hold the prompt and response by default, and cf-aig-collect-log-payload: false keeps the metadata while dropping the bodies. Vercel’s request logs show every routing attempt for 30 days, a generation lookup API returns provider, latency, tokens and cost by ID, and trace drains forward an OpenTelemetry trace per request over OTLP/HTTP with one span per provider attempt, using gen_ai.* and vercel.ai_gateway.* attributes; traces carry metadata but not prompt or completion content, and cost $0.05 per 1,000 traces plus $0.50 per GB on Pro and Enterprise plans.

Deployment and data residency

This is the constraint that most often decides the choice on its own. A hosted gateway means prompts and completions transit a third party’s infrastructure; a self-hosted gateway keeps everything inside your boundary at the cost of running it.

Bifrost runs as a binary or container anywhere, with Helm and Kubernetes guides in the docs; the enterprise tier adds in-VPC deployment, SAML SSO, RBAC, vault integrations (HashiCorp, AWS Secrets Manager, Google Secret Manager, Azure Key Vault) and clustering. The cluster uses memberlist gossip on port 10101 (TCP and UDP) for membership and a gRPC channel on 10102 for application state, replicating governance usage counters, configuration, routing rules, virtual keys, providers, RBAC, MCP tools and pricing with eventual consistency the docs describe as converging “within seconds”; three nodes are the recommended minimum to tolerate one failure, and single-node clusters are unsupported in production. LiteLLM ships containers, Helm charts and Terraform modules for ECS Fargate and Cloud Run, and needs PostgreSQL plus Redis for a multi-instance deployment. Kong runs self-managed in traditional, hybrid or DB-less mode, or with a Konnect control plane and data planes in your environment; the cluster strategy for AI rate limiting is unsupported in hybrid and Konnect modes, so Redis is the practical choice there.

Cloudflare and Vercel cannot be self-hosted. Vercel’s answer to residency is regional inference: inferenceRegion pins the provider’s inference and any at-rest storage to us or eu, fails with a 400 rather than falling back out of region, and reports the resolved region in the response; team-wide zero data retention and provider allowlists are add-ons at $0.10 per 1,000 requests. The gateway itself deletes prompts and responses after each request by default. Cloudflare’s documentation describes log payload controls and DLP scanning; we did not find a region-pinning control for inference in the pages we read.

Licensing and the cost of ownership

The sticker price of a gateway is rarely the cost. What matters is which features sit in the free tier, what you have to run alongside it, and what a hosted service charges on top of tokens.

Two things stand out. First, Kong’s free surface is thinner than its plugin count suggests: the pages for AI Proxy Advanced, AI Rate Limiting Advanced, AI Semantic Cache, AI Semantic Prompt Guard and AI MCP Proxy each state that the plugin “is only available as part of our AI Gateway Enterprise offering”, while ai-proxy and ai-prompt-guard carry no such marker. If you already pay for Kong Enterprise or Konnect this is moot; if you were hoping to use Kong OSS as an AI gateway, it is the whole story. Second, Bifrost and LiteLLM both keep the operational core, including budgets, virtual keys and MCP, in the open-source tier and charge for the things a security team asks for: SSO, RBAC, audit and guardrail scoping. The real cost of an AI product is dominated by tokens and people, and a gateway’s licence is a rounding error next to either; the operational cost of running PostgreSQL and Redis, or of a hosted hop on every call, is not.

Two teams, two decisions

A team running an agent platform in a regulated bank. Requests carry customer data, so nothing may leave the VPC; volumes are high because every agent step is a model call; the security team wants SSO, audit logs and the ability to block a tool call. The hosted gateways are out on the first constraint. Kong is plausible if the bank already runs Kong Enterprise, and its declarative policy model will please the platform team, but the AI load balancer, rate limiter and MCP proxy all sit in the enterprise tier and the semantic features need Redis or pgvector. LiteLLM works, with PostgreSQL and Redis alongside and one Uvicorn worker per pod scaled horizontally. Bifrost is the fit we would argue for: a single binary in the VPC, the fallback chain and key rotation handle provider 429s without application code, virtual keys give each agent a budget, the enterprise tier adds clustering, SSO, RBAC, audit logs and guardrails that can block an MCP tool execution, and the MCP gateway means the agents’ tool servers are governed by the same keys as their model calls.

A three-person startup on Vercel shipping a consumer chat product. Volume is modest, there is no compliance requirement yet, and nobody wants to run infrastructure. Vercel AI Gateway costs nothing on tokens, gives provider failover and per-key budgets, and traces every attempt; Cloudflare is the equivalent if the stack is already on Workers, with the caveat that rate limiting is one number per gateway and guardrails cannot be used on streamed responses. The moment this team signs an enterprise customer with residency requirements, Vercel’s inferenceRegion covers the provider side but not where the gateway itself runs, and a move to a self-hosted gateway becomes the conversation. That migration is cheap if the application has only ever talked to an OpenAI-compatible endpoint, which is a reason to use one from the start.

Reading the numbers

Bifrost’s figures are the most complete published set in the category: overhead at two instance sizes, a same-hardware head-to-head on real OpenAI traffic, and a public harness to reproduce both. Go versus Python explains the direction of the gap, since a compiled gateway with no interpreter or database on the request path does less work per call. Kong, Cloudflare and Vercel publish nothing comparable, and hosted gateways add a network hop that depends on where the application runs. Feature claims are read from documentation, not tested, and documentation changes: Cloudflare’s Universal Endpoint went from headline feature to deprecated within the life of this article’s first edition. The practical step is the one that applies to model benchmarks too: run the published harness on your own instance sizes and traffic before the number decides anything.

Figure 3: One constraint usually settles the choice before any feature comparison matters.

Recommendations keyed to constraints

• You have data-residency or air-gap requirements. Self-host. Choose Bifrost if throughput, memory and explicit error semantics matter and you want a single binary with governance and MCP built in; choose LiteLLM if your team is Python-first, already runs PostgreSQL and Redis, and values its provider breadth and YAML familiarity.
• You already run Kong. Kong AI Gateway extends the platform you have, and consistency across your API estate usually outweighs feature-by-feature comparison. Budget for the enterprise tier, because load balancing, failover, AI rate limiting, semantic caching and the MCP proxy all live there.
• You want zero operations and are on Vercel. Vercel AI Gateway, with budgets at every scope, per-attempt routing traces and no token markup. Set order and models explicitly rather than relying on the dynamic default if you care which account absorbs a 429.
• You want zero operations and are on Cloudflare. Cloudflare AI Gateway for caching, analytics and Dynamic Routing. Accept a single per-gateway rate limit and unstreamed guardrails, and plan for Workers Paid if you need Logpush.
• You run agents at scale, on any infrastructure. Bifrost. The per-request overhead, the retry budget per fallback provider, the key rotation on 429 and the MCP gateway with per-key tool filtering are the four things an agent platform exercises hardest, and Bifrost documents each precisely. Validate the vendor’s throughput numbers on your own instance size first, as its own benchmark page advises.
• You are undecided. Put an OpenAI-compatible endpoint in front of your application today and defer the gateway choice. All five speak that dialect, and re-benchmarking models quarterly is where the gateway’s real savings come from, whichever one you pick.

The judgement

We think the category has split into two products that share a name. One is a control plane for model access, where the questions are error semantics, budgets, guardrails, tool governance and where the bytes go; Bifrost and LiteLLM compete here, Kong joins them if you pay, and Bifrost’s combination of explicit failover classes, governance depth and an MCP gateway in the open-source tier makes it the reference point as of September 2026. The other is a convenience layer for teams that would rather not run anything; Vercel and Cloudflare compete there, and Vercel’s budgets and routing traces make it the more complete of the two. Decide which product you are buying before comparing features, and hold every benchmark, including the ones quoted here, to the conditions it was measured under.

Feature claims are drawn from each vendor’s public documentation and published benchmarks as of September 2026. Verify current specifics before deciding.