Top 5 AI gateways for failover across Bedrock, Vertex AI and Azure OpenAI in 2026

Amazon Bedrock quotas, Vertex AI throughput tiers and Azure OpenAI tokens-per-minute limits are three independent ceilings on what is often the same model: Claude is served by Bedrock and Vertex AI, and GPT models by Azure OpenAI and OpenAI itself. Hitting one ceiling returns a 429 even while the identical model has spare capacity on another cloud. This ranking is for platform teams that buy model capacity from more than one hyperscaler and want an AI gateway to fail over between them automatically, by error class, without each application carrying three SDKs and three retry loops.

The difficulty is that the three clouds disagree on almost everything except the model weights. Bedrock signs every request with AWS Signature Version 4, Vertex AI wants OAuth2 tokens minted from a Google service account, and Azure OpenAI addresses models by deployment name inside a resource, authenticated by an API key or Microsoft Entra ID. Each meters capacity differently, each sells reserved throughput differently, and each words its throttling errors differently. A gateway earns its place here only if it absorbs all of that and still makes the right decision when a request fails.

We ranked five gateways on that job. Every product claim below comes from the vendor’s own documentation, and every cloud quota figure from AWS, Google or Microsoft documentation, read on 9 and 10 October 2026.

Why the same model hits three different walls

Each cloud meters model capacity with its own unit, scope and reservation product, so a workload can be throttled on one cloud while its quota on another sits unused. That asymmetry is the reason multi-cloud failover pays off, and the details below decide how a gateway should react to each cloud’s errors.

Amazon Bedrock quotas

AWS’s quotas page for Bedrock says model inference is “controlled by quotas on token usage”, applied per account and tracked separately for the bedrock-runtime and bedrock-mantle endpoints even when both call the same model. Two details matter for failover planning. The quotas applied to an account “might be lower than the default values” depending on account age, usage and payment history, and increases “aren’t granted automatically”. A new AWS account, or a new Region in an old one, may start well below the published default.

When an account crosses a quota, Bedrock returns HTTP 429 with a ThrottlingException. The Bedrock error-code reference separates that from two other capacity errors: a 503 ServiceUnavailable, which AWS says “is not related to your account-level quotas”, and a 529 overloaded_error, described as “a transient capacity error”. The fixes differ. A 429 means the account is out of allowance; a 503 or 529 means the service is short of capacity, and retrying with backoff and jitter is the documented response.

AWS offers two tools of its own. Cross-Region inference routes a request to another Region through an inference profile, either within a geography (US, EU, APAC) or globally, with global profiles priced “approximately 10% savings” below standard and no extra routing charge. Provisioned Throughput reserves model units billed hourly, with no-commitment, one-month or six-month terms. The two do not combine: AWS states that “inference profiles currently don’t support Provisioned Throughput”.

Vertex AI quotas and 429s

Google’s documentation now files Vertex AI’s generative models under the Gemini Enterprise Agent Platform name, but the API and quota model are the same. Standard pay-as-you-go uses usage tiers keyed to an organisation’s spend over a rolling 30 days. For Gemini Pro models, Tier 1 ($10 to $250) aims at 500,000 tokens per minute at organisation level and Tier 4 (over $50,000) at 10,000,000. Those baselines assume the global endpoint; Google says single-region endpoints such as us-central1 “have the lowest capacity headroom”, and that sharp second-level spikes can be throttled “even if your average per-minute usage is below your limit”.

The 429 reference shows the message depends on how capacity was bought: “Resource exhausted, please try again later” on pay-as-you-go, and “Too many requests. Exceeded the Provisioned Throughput” on a Provisioned Throughput subscription, which reserves capacity in units Google calls GSUs. By default, overage above a subscription is served as pay-as-you-go rather than rejected. Claude on Google Cloud follows the same two purchase paths, pay-as-you-go or provisioned throughput.

TPM quota and PTUs for GPT deployments

Microsoft’s quota guide assigns tokens-per-minute quota “per-region, per-model, per-deployment-type”. Its own example: 240,000 TPM for GPT-4o in East US can be one 240K deployment, two 120K deployments, or any split across resources in that region. A requests-per-minute limit follows from the TPM figure; for older chat models one capacity unit is 6 RPM per 1,000 TPM.

Provisioned throughput units are model-independent but region-specific, so PTU quota in East US does not carry over to another region. Spillover sends overflow from a saturated provisioned deployment, for example “a 429 when PTUs are exhausted”, to a standard deployment, but only one in the same Foundry resource. Deployment types add another axis: Global, Data Zone and regional variants differ in where data is processed and how much quota they carry, and Microsoft recommends Global Standard as the starting point.

Figure 1: The same model family sits behind three authentication schemes and three quota systems, which is what a multi-cloud gateway has to hide.

What none of the clouds does

All three clouds offer a way to absorb overflow inside their own walls: cross-Region profiles on Bedrock, the global endpoint on Vertex AI, spillover on Azure. None of them fails over to a competitor. If a team’s Bedrock quota for Claude is exhausted in the middle of a launch, Bedrock will not send the request to the Claude quota the same team holds on Vertex AI, and Azure will not send a GPT request to OpenAI. That last hop belongs to whatever sits in front of the clouds, which is the job this ranking evaluates.

How the five gateways were ranked

The ranking weights seven criteria, each read from primary documentation. The first three carry the most weight, because a gateway that cannot authenticate natively to all three clouds or cannot tell a 429 from a 400 will fail over at the wrong moments.

Readers new to the category can start with the site’s explainer on what an AI gateway is. The broader comparison of five LLM gateways covers governance and MCP; this piece narrows to multi-cloud failover.

The five gateways at a glance

1. Bifrost

Bifrost is an open-source AI gateway written in Go by Maxim AI, released under Apache 2.0 with a paid enterprise tier. It exposes one OpenAI-compatible API over 25+ providers and 10,000+ models, and Bifrost’s published benchmark reports 11 µs of added overhead per request at 5,000 RPS on an AWS t3.xlarge. It gets the longest write-up here because its documentation answers every criterion in the table above, including the three-cloud authentication details most gateways leave to the reader.

Three clouds, three native auth paths. The Bedrock provider page documents automatic SigV4 signing with four flows: explicit access keys, the inherited AWS credential chain (IRSA on EKS, ECS task roles, EC2 instance profiles), STS AssumeRole with an external ID, and Bedrock’s bearer-token API keys. It also maps model names to cross-Region profile IDs or application inference profile ARNs, so a us. profile and a Region-pinned model can live on different keys. The Vertex AI page covers service account JSON with automatic token refresh and Application Default Credentials (including GKE Workload Identity), and routes Gemini and Claude models to the right Vertex endpoint by model name. The Azure page supports API keys, Entra ID service principals and DefaultAzureCredential for managed identity, with aliases that map a model name to an Azure deployment ID.

Errors classified before anything is retried. The retries and fallbacks documentation splits failures into per-key failures and transient server failures. A 429 marks the key as used for that cycle and rotates to another key, but still applies backoff “to let account-level quota windows slide”, which suits Bedrock and Azure, where quota is shared across keys in one account or region. A 401, 402 or 403 marks the key dead for that request and rotates with no wait. A 5xx or network error retries the same key with exponential backoff and jitter (500 ms initial, 5 s cap by default). A 400, 404 or 422 is returned to the caller without retrying. If every key is dead, the gateway returns 502 upstream_credentials_exhausted rather than the raw 401, so callers do not mistake a broken cloud credential for a broken gateway key.

Fallback chains across clouds. When a provider exhausts its retries, Bifrost moves to the next entry in a fallbacks list of provider/model strings, and each fallback gets “its own full retry budget”. A Claude request can therefore try Bedrock, then Vertex AI, then Anthropic’s API directly, with the serving provider reported in the response’s extra_fields.provider. Through virtual-key routing, the chain can be generated automatically: providers configured on a virtual key are weighted for load balancing and, when the request carries no explicit fallbacks, sorted by weight into a fallback chain. The same page notes cross-provider routing is never implicit; a model must be allowed on each provider that may serve it.

Pooling keys and Regions. Key management uses weighted random selection across keys, with per-key model allowlists and denylists and deployment mapping for Azure and Bedrock. Because each Bedrock or Vertex key carries its own Region, two keys for the same model in us-east-1 and us-west-2 act as a regional pool before any cross-cloud fallback is needed. One Azure-specific detail stands out: Azure streams can report a rate-limit error inside an HTTP 200 after sending startup metadata, and the documentation says Bifrost buffers those startup events so the error still reaches retry and fallback logic.

Health tracking and PTU spillover. Adaptive load balancing recomputes route weights every five seconds from error rate, latency and utilisation, moves routes through Healthy, Degraded, Failed and Recovering states, and shares TPM rate-limit signals across nodes so an overloaded key is backed off fleet-wide. The circuit breaker trips on provider response headers; its documented example reroutes from an Azure PTU deployment to a pay-as-you-go deployment when Azure returns X-Ms-Is-Spilled-Over: true, and it can read the cooldown from a header such as retry-after-ms.

Enterprise tier. Adaptive load balancing, the header-based circuit breaker, clustering, SSO, RBAC and audit logs come with Bifrost Enterprise.

Best for: teams that hold capacity on two or three hyperscalers and want one self-hosted gateway that authenticates natively to each and fails over by error class.

Figure 2: Retries stay inside one provider until their budget runs out; only then does the request cross to another cloud.

2. LiteLLM

LiteLLM is an open-source Python SDK and proxy from BerriAI with the broadest provider list among open-source gateways, and its router is the most configurable failover engine in this ranking after Bifrost. It authenticates to all three clouds: the Bedrock provider accepts access keys, aws_role_name, web identity tokens and Region settings; the Vertex AI provider takes service account credentials and a location; and the Azure provider supports API keys and Entra ID tokens.

Strengths. The reliability documentation offers three fallback types beyond the generic one: context_window_fallbacks, content_policy_fallbacks and default_fallbacks. Deployments in one model group can carry an order, so order=1 deployments are tried before order=2, and on a 429 the failed deployment “is immediately placed on cooldown”. allowed_fails and cooldown_time take a misbehaving deployment out of rotation for a set period. The routing page adds a RetryPolicy with a separate count per exception type (RateLimitErrorRetries, AuthenticationErrorRetries, TimeoutErrorRetries and others), plus routing strategies from simple shuffle to latency-based, usage-based and lowest-cost.

Trade-offs. Multi-instance deployments need Redis to share cooldowns and TPM/RPM usage, and the routing page warns that usage-based routing “adds significant latency due to Redis operations”. The generic fallback list “covers all errors (429, 500, etc.)”, so error-class nuance comes from combining retry policy, cooldowns and order rather than from one built-in classification. The site’s guide to LiteLLM alternatives covers the operational side in more detail.

Best for: Python-first platform teams that want fine-grained, per-exception control over retries and are prepared to run Redis alongside the proxy.

3. Kong AI Gateway

Kong AI Gateway is a set of AI plugins on Kong Gateway, and the AI Proxy Advanced plugin is where multi-cloud failover lives. Its provider list includes Azure OpenAI, Amazon Bedrock, Gemini and Vertex AI, and Kong’s AI provider FAQ covers Bedrock cross-Region inference profiles, IAM credentials or assumed roles for Bedrock, and managed identity for Azure when the gateway runs on Azure.

Strengths. The balancer offers six algorithms: round-robin, consistent hashing, least connections, lowest latency, semantic routing and priority, the last described as “tiered failover across model groups”. From Kong Gateway 3.10, fallback “works across targets with any supported format”, so a Bedrock target can fall back to Vertex AI or Azure. From 3.13, a circuit breaker removes a target after max_fails failures for fail_timeout. For organisations that already run Kong for their APIs, AI traffic inherits the same authentication, rate limiting and logging plugins.

Trade-offs. Kong states that “client errors don’t trigger failover”, and failing over on a 429 or 502 requires adding http_429 or http_502 to config.balancer.failover_criteria, plus non_idempotent for POST requests. That is a sensible conservative default, but it means quota exhaustion does not fail over until someone configures it. AI Proxy Advanced is “only available as part of our AI Gateway Enterprise offering”, and its semantic features need Redis with the Query Engine or Valkey.

Best for: organisations already standardised on Kong Gateway Enterprise that want multi-cloud AI failover under the same policy model as their other APIs.

4. Azure API Management AI gateway

Azure API Management’s AI gateway capabilities cover language model APIs that follow the OpenAI, Anthropic Messages or Google Vertex AI schemas, and Microsoft explicitly includes models hosted on non-Microsoft providers such as Amazon Bedrock. A unified model API, in preview, exposes several backends through one endpoint.

Strengths. The backend pool load balancer supports round-robin, weighted, priority-based and session-aware distribution, which suits the common pattern of a PTU deployment at priority 1 and pay-as-you-go deployments behind it. The circuit breaker uses a “dynamic trip duration, applying values from the Retry-After header”, and backends authenticate to Azure OpenAI through system- or user-assigned managed identity. Token-limit and token-metric policies sit alongside, so quota enforcement and failover live in one service that Azure-centric teams already operate.

Trade-offs. Reaching Bedrock is a manual job. The Bedrock passthrough guide stores long-term IAM user access keys as named values and computes the SigV4 signature in a C# policy expression, which Microsoft itself cautions should be managed “as securely as you would a password”. When a circuit trips, API Management returns a 503 to the client for the trip duration, tripping rules are “approximate” because gateway instances do not synchronise, and the circuit breaker is not available on the Consumption tier. The documentation does not describe distinct handling for 401 versus 429 versus 5xx.

Best for: Microsoft-standardised enterprises whose primary capacity is Azure OpenAI PTUs and who need Bedrock or Vertex AI only as an occasional overflow target.

5. Cloudflare AI Gateway

Cloudflare AI Gateway is a hosted gateway that proxies Amazon Bedrock, Google Vertex AI and Azure OpenAI, each through a provider-specific URL that mirrors the cloud’s own path.

Strengths. Nothing to deploy. For Bedrock, AWS credentials stored as bring-your-own-key provider keys let the gateway sign each request automatically. Fallbacks are defined as an ordered array on the Universal endpoint, triggered by errors or request timeouts, with a cf-aig-step response header showing which step answered. Request handling adds up to five retry attempts with constant, linear or exponential backoff, and Dynamic Routing builds versioned flows with rate-limit and budget nodes that switch to a fallback model when exceeded.

Trade-offs. Fallback fires “if a model request returns an error”, with no documented distinction between a 429 and a 400, so a malformed request can be replayed against every provider in the chain. Bedrock BYOK relies on stored accessKeyId and secretAccessKey rather than IAM roles, the Vertex AI page asks for a service account JSON or access token, and the Azure OpenAI page documents API-key authentication. As a hosted service it cannot run inside the VPC where those credentials normally stay.

Best for: teams already on Cloudflare Workers that want multi-cloud fallback without operating a service and can accept static cloud credentials.

What actually differs between the five

All five can put Bedrock, Vertex AI and Azure OpenAI behind one endpoint. They differ on three questions that decide whether failover helps or makes an incident worse.

Figure 3: Failover that ignores the status code either gives up too early on a 429 or replays a bad request across every cloud.

Error handling

A multi-cloud setup multiplies the cost of bad error handling. Retrying a 400 against three clouds triples the wasted calls, and failing over on a 401 without rotating keys burns the fallback budget on a credential problem. Bifrost publishes a full table of which status codes retry, rotate keys or back off. LiteLLM gets close through per-exception retry counts and 429 cooldowns. Kong is conservative by default and fails over on 429 only when configured. API Management relies on circuit breaking and priority pools, and Cloudflare treats any error the same way.

Credentials

Bedrock is the hardest of the three to front because SigV4 signs each request with a derived key, and the AWS signing reference leaves little room for shortcuts. Gateways that inherit IAM roles (Bifrost, LiteLLM, Kong) let the credential stay short-lived and scoped to the workload. Gateways that store access keys (Cloudflare BYOK) or compute signatures in policy (API Management) move a long-term AWS secret into another system. The same pattern repeats on Azure, where managed identity beats a stored API key, and on Vertex AI, where ADC and Workload Identity beat a pasted service account file.

Reserved capacity

Teams that pay for PTUs or Provisioned Throughput want those units saturated before pay-as-you-go traffic starts, and they want overflow to land somewhere sensible. API Management’s priority pools and Bifrost’s header-based circuit breaker both model that explicitly for Azure. On Vertex AI the overflow is automatic, since overage defaults to pay-as-you-go, and on Bedrock the gateway has to treat a Provisioned Throughput model ARN and a cross-Region profile as separate targets, because AWS does not let the two combine.

Recommendations by constraint

The right gateway follows from which cloud holds most of the capacity and where credentials are allowed to live. The list below maps common situations to a first choice; teams comparing more general gateway features should read the site’s ranking of ten AI gateways alongside this one.

• Claude on both Bedrock and Vertex AI, GPT on Azure and OpenAI. Bifrost. It authenticates natively to all four providers, classifies the error before acting, and walks a cross-cloud fallback chain without code in the application.
• Credentials must never leave the VPC. Bifrost or LiteLLM, self-hosted, using IAM roles, Workload Identity and managed identity rather than stored keys.
• Fine-grained retry counts per exception type in Python. LiteLLM, with Redis provisioned for shared cooldown state.
• Kong Gateway Enterprise already in production. Kong AI Proxy Advanced, with http_429 added to failover_criteria on day one.
• Mostly Azure OpenAI PTUs, other clouds as rare overflow. Azure API Management priority pools, accepting manual SigV4 for any Bedrock backend.
• No infrastructure to operate and already on Cloudflare. Cloudflare AI Gateway, with fallback chains kept short because every error triggers them.

For readers who want a second view of the category, Maxim AI publishes a production-ready comparison of LLM gateways, and its LLM gateway overview describes how Bifrost positions routing and failover; both are vendor sources and should be read as such.

Questions to settle before turning failover on

Three decisions are easier to make before an incident than during one:

1. Is the fallback model equivalent? Claude Sonnet on Bedrock and on Vertex AI carries the same name, but feature support can differ by cloud; Bifrost’s Vertex page, for example, lists which anthropic-beta headers Vertex accepts and drops the rest. Test tool use, long context and streaming on every fallback target.
2. Does data residency allow the hop? A fallback from an EU Bedrock profile to a global Vertex endpoint can move processing outside the EU. AWS and Microsoft both separate geographic from global deployment types for this reason; the gateway’s chain should respect the same boundary.
3. Who sees the failover? Log which provider served each request. Bifrost returns it in extra_fields.provider, Cloudflare in cf-aig-step. Without that, a silent shift of half the traffic to a more expensive cloud shows up only on the invoice.

Verdict

Multi-cloud failover is worth the setup only when the gateway reads the error before it acts, signs requests the way each cloud expects, and treats reserved capacity as something to fill before spilling. On those terms Bifrost ranks first: its open-source tier documents per-status-code behaviour, native SigV4, Vertex token and Entra ID authentication, regional key pools and cross-cloud fallback chains, and its enterprise tier adds health-aware weighting and a circuit breaker built around Azure’s spillover header.

LiteLLM is a capable second for Python shops willing to run Redis. Kong and Azure API Management make sense when their wider platforms are already paid for, provided someone configures 429 failover in Kong and accepts hand-signed Bedrock calls in API Management. Cloudflare AI Gateway is the easiest to start with and the least discriminating once something fails. We think the clouds will keep adding their own overflow paths, cross-Region profiles, global endpoints and spillover among them, but none of them will route a throttled request to a competitor’s copy of the same model, and that hop is what the gateway is for.