Top 5 AI gateways for air-gapped and in-VPC LLM deployments in 2026

An on premise LLM deployment keeps model inference, prompts and responses inside infrastructure the organisation controls, and the AI gateway is the component that decides whether that promise holds. The gateway holds the provider credentials, sees every prompt in plain text, writes the request logs and chooses which model endpoint a call goes to. If it phones home, depends on a vendor’s SaaS control plane, or quietly downloads data at startup, the rest of the private stack inherits that exposure.

This matters most to defence contractors, hospitals, banks and public-sector agencies, where control frameworks such as NIST SP 800-53 and FedRAMP expect boundary protection, audit trails and protected credentials, and where some enclaves have no internet route at all. This ranking compares five self-hosted AI gateways on the things those teams actually check: how they install offline, what they call out to by default, how they reach private model endpoints, where secrets live and what evidence they produce for auditors.

What “inside the network” has to mean

A gateway qualifies for an air-gapped or in-VPC deployment only if it can run with no dependency on the vendor’s infrastructure after installation. Four properties decide that: an offline install path, no outbound calls you did not configure, no SaaS control plane on the request path, and support for private model endpoints.

Offline installation. Air-gapped clusters cannot pull from Docker Hub. The gateway must ship as a container image and, ideally, a Helm chart that can be mirrored into an internal registry, pinned by digest and pulled locally. Kubernetes’ own image documentation explains that the default IfNotPresent pull policy skips pulling an image that already exists on the node, which is the behaviour a pre-loaded or mirrored enclave relies on.

No unconfigured outbound calls. Usage analytics, update checks, remote pricing files and licence validation all count. In an air-gapped enclave they fail; in an in-VPC deployment they open an egress path that the security team has to justify. NIST SP 800-53 groups these concerns under boundary protection (SC-7) and audit and accountability (the AU family), and NIST SP 800-207 on zero trust assumes every flow is explicitly authorised.

No SaaS control plane. Several API gateways split a hosted control plane from self-hosted data planes. That is a sensible design for most companies, but in a disconnected enclave the control plane has to run locally too.

Private model endpoints. In-VPC teams usually mix self-hosted models (vLLM, Ollama, SGLang) with cloud models reached privately. Amazon Bedrock supports interface VPC endpoints powered by AWS PrivateLink, so instances need no public IP, internet gateway or NAT device to reach it. Azure OpenAI supports private endpoints, and setting public network access to Disabled makes the private endpoint the only way in. A private endpoint is a network interface with a private IP address in your virtual network, so the gateway calls a hostname that resolves inside the VNet.

These properties exclude hosted gateways by definition. Cloudflare AI Gateway runs on Cloudflare’s network and Vercel AI Gateway is called at ai-gateway.vercel.sh. Both are good products for teams without residency constraints, and both are covered in the site’s broader ranking of AI gateways in 2026, but neither can sit inside a VPC or an enclave.

Figure 1: In a private deployment every arrow stays inside the boundary, including the gateway’s own telemetry, secrets and logs.

How we ranked the five

The criteria come from what a security review of a private LLM stack asks for, weighted towards the properties that cannot be bolted on later. Each tool was assessed from its own documentation as of October 2026; GenAI Brief did not run the gateways for this piece.

Feature breadth carries only 10 per cent because every gateway here routes requests well; the reason teams fail an accreditation review is usually a dependency they did not know about. Readers who want the full self-hosted field, rather than the subset that suits regulated enclaves, can start with the site’s ranking of open-source AI gateways you can self-host.

The five at a glance

1. Bifrost

Bifrost is an open-source AI gateway written in Go and released under Apache 2.0 by Maxim AI, with its code on GitHub. It exposes 25+ providers and 10,000+ models through one OpenAI-compatible API, and it is the one gateway in this ranking whose documentation addresses every criterion above, which is why its entry runs longer than the others.

Offline install. The deployment requirements describe a single Linux container image, docker.io/maximhq/bifrost, published for linux/amd64 and linux/arm64, running as user 1000 on port 8080. The docs tell operators to pin a version or image digest, which is exactly what a mirrored registry needs. The official Helm chart covers both the open-source and enterprise builds, and the Helm guide includes a self-hosted registry option for the image pull secret. Enterprise customers receive images from a private registry, which they can mirror into an internal one.

Outbound behaviour. The deployment requirements list the outbound access Bifrost needs: DNS and network routes to the model providers, PostgreSQL, MCP servers, vector stores, identity systems and observability endpoints that the operator configures. In Bifrost’s documentation, “telemetry” means the Prometheus metrics the gateway serves at GET /metrics on its own HTTP port, for an internal scraper to collect; the plugins reference describes an optional push to a Pushgateway the operator runs. The model catalog syncs a pricing sheet for cost tracking, and its pricing_url and sync interval are configurable, so an enclave can serve that file from an internal host.

Private model endpoints. The Helm provider guide covers self-hosted providers (Ollama, vLLM and SGLang) by URL with no API key, for example a vLLM service at a cluster-local address. For cloud models it documents IRSA and EKS Pod Identity or STS AssumeRole for Bedrock, Managed Identity and AKS Workload Identity for Azure OpenAI through DefaultAzureCredential, and GKE Workload Identity for Vertex AI. That removes static cloud keys from the configuration entirely, and PrivateLink or Private Link DNS then keeps the traffic on private addresses.

Hardening and storage. The security page describes production images built on a FIPS 140-2 validated Alpine base, running as a non-root user, with statically linked, stripped binaries; Docker Scout scans every image pushed to Docker Hub, and GitHub Actions are pinned to commit SHAs. The open-source build stores configuration in SQLite or PostgreSQL 16+, and logs in SQLite, PostgreSQL or ClickHouse, all of which can run inside the boundary.

Governance that runs locally. Virtual keys carry budgets and rate limits per application or team, and retries with provider fallbacks handle failing endpoints, both in the open-source tier. In the enterprise tier, the in-VPC deployment offering targets GKE, EKS and AKS clusters inside the customer’s VPC, installed with the provided Helm charts.

Secrets, audit and identity. Secret management connects HashiCorp Vault (KV v2, token or AppRole auth), AWS Secrets Manager or GCP Secret Manager, so the database stores a vault.<path> reference instead of a plaintext key, and the default read_only mode never writes to the vault. Audit logs record administrative changes, can be signed with an HMAC key, have configurable retention and export as JSON, JSON Lines or Syslog for a SIEM. Identity federation supports self-hosted Keycloak and Microsoft Entra ID including GCC High and DoD clouds.

Guardrails without cloud calls. Guardrails include in-process custom regex with no external service dependency and a Presidio integration for a privately deployed PII analyser, which matters in an enclave where cloud moderation APIs are unreachable. For background on what AI guardrails check and where they sit in the request path, the vendor maintains an overview.

Performance. Bifrost’s published benchmark reports 11 µs of added overhead per request at 5,000 RPS on an AWS t3.xlarge. In a private deployment, where the gateway often fronts GPU servers that are the expensive, scarce resource, a thin proxy layer keeps capacity planning about the models.

Enterprise tier. In-VPC deployment support, clustering, secret-manager integration, RBAC, SSO and audit logs come with Bifrost Enterprise.

Best for: regulated teams that want one gateway with an offline-friendly install, local telemetry, private model routing and audit-grade controls, without assembling those properties from several products.

2. LiteLLM

LiteLLM is BerriAI’s Python proxy and SDK, an open-source gateway with an OpenAI-compatible interface across a long list of providers. It is a credible choice for private deployments and states its self-hosting posture plainly.

Strengths. The data security page states that “we run no telemetry when you self-host LiteLLM” and that no personal data is transmitted to LiteLLM servers by self-hosted instances. The deployment guide supports Helm on EKS, GKE and AKS and official Terraform modules on AWS and GCP, and notes that all images are signed, with a non-root variant. For Bedrock, the aws_bedrock_runtime_endpoint parameter points calls at a specific runtime endpoint, which suits VPC endpoint setups. The Enterprise licence is a key on the same image, and LiteLLM describes it as running “in your cloud, so prompts and responses never leave your environment.”

Trade-offs. Two outbound defaults need changing for an enclave. The configuration reference explains that the model cost map is fetched from GitHub at startup unless LITELLM_LOCAL_MODEL_COST_MAP is set to True, and a separate LITELLM_LOCAL_BLOG_POSTS flag disables remote fetching of blog content. The Helm path needs PostgreSQL and Redis inside the boundary. Secret managers (AWS KMS and Secrets Manager, Azure Key Vault, CyberArk Conjur, Google Secret Manager and KMS, HashiCorp Vault), SSO, SCIM, audit logs and IP allowlists all require the Enterprise licence. Whether licence activation needs network access is not published on the pages reviewed. Teams weighing a move away from LiteLLM can compare options in the site’s guide to LiteLLM alternatives.

Best for: Python-first platform teams that want the broadest provider coverage and are prepared to run PostgreSQL and Redis in the enclave and to flip the offline flags before go-live.

3. Kong AI Gateway

Kong AI Gateway is a set of AI plugins on Kong Gateway, which many regulated organisations already operate for REST APIs. Its fit for a private deployment depends heavily on which Kong topology is in play.

Strengths. Kong’s deployment topologies include hybrid mode, traditional mode and DB-less mode, and hybrid mode works on both Konnect and self-managed Kong Gateway. In hybrid mode, “if a Control Plane is offline, Data Planes will run using their last known configuration”, and a compromised data plane cannot affect other nodes. The AI Proxy plugin supports OpenAI, Azure OpenAI, Amazon Bedrock, Anthropic, Gemini, Vertex AI, Mistral, Hugging Face, Ollama, Databricks and vLLM, among others.

Enterprise controls. Vaults in Kong Gateway Enterprise integrate AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager, HashiCorp Vault and CyberArk Conjur, and can even hold the Kong licence itself. Admin API audit logs record requests and database changes and can be signed with an RSA key.

Trade-offs. Konnect is Kong’s hosted control plane; Kong’s own vault documentation notes that secrets in a Konnect Config Store resolve only after the gateway connects to the control plane. A fully disconnected deployment therefore means self-managed Kong Gateway, in which case the audit log feature applies (Kong lists it as incompatible with Konnect). On the open-source build, the only vault backend is environment variables; every external secret manager is an Enterprise feature. The configuration reference shows anonymous_reports, which sends anonymous usage data such as error stack traces to Kong, defaults to on, so it must be set to off. The AI Proxy Advanced plugin, which adds load balancing across models, is marked “AI License Required”.

Best for: organisations that already run self-managed Kong Gateway Enterprise and want AI traffic under the same policies, consumers and audit trail as their REST APIs.

4. Apache APISIX AI gateway

Apache APISIX is an Apache Software Foundation API gateway with AI plugins for proxying, multi-provider load balancing, token rate limiting and prompt guarding. Everything is open source, which removes licence-key logistics from an enclave.

Strengths. APISIX documents three deployment modes. In standalone mode the data plane loads configuration from a local YAML or JSON file, so a disconnected node needs neither etcd nor an Admin API, and configuration can arrive through the same transfer process as the software. Installation paths include Docker, Helm charts and an RPM repository. The ai-proxy-multi plugin supports OpenAI, DeepSeek, Azure OpenAI, Anthropic, Gemini, Vertex AI, Amazon Bedrock and any OpenAI-compatible API, and its override.endpoint field replaces the default provider endpoint, which is how a vLLM server or a private endpoint hostname is wired in. Secrets can come from environment variables, HashiCorp Vault, AWS Secrets Manager or GCP Secret Manager in the open-source build. The ai-prompt-guard plugin applies allow and deny patterns to prompts locally.

Trade-offs. APISIX publishes no gateway-level statement on telemetry, so the security team will want to verify egress itself. There is no budget hierarchy or virtual-key model comparable to the purpose-built gateways, admin audit logging is not described in the AI gateway documentation, and guardrails beyond pattern matching rely on external services that may be unreachable in an enclave.

Best for: teams that already run APISIX, or that want an entirely open-source gateway they can configure from files and carry across an air gap.

5. Agent Router (formerly Envoy AI Gateway)

Agent Router is the project previously called Envoy AI Gateway, now part of the Agentic AI Foundation, with “same code, same maintainers” according to its documentation. It is a control plane on Envoy Gateway that manages providers, credentials and routing as Kubernetes resources.

Strengths. Installation uses Helm charts published as OCI artefacts on Docker Hub (a CRD chart followed by the controller), which mirror into an internal OCI registry like any image. An aigw run command runs the gateway as a standalone proxy outside Kubernetes, and the project’s Docker Compose examples include an Ollama-backed stack and an OpenTelemetry stack for local traces and metrics. The supported-providers table lists self-hosted models that speak the OpenAI schema, naming vLLM as an example, alongside Bedrock and Azure OpenAI. For cloud providers, upstream authentication uses OIDC with AWS STS to generate temporary credentials for Bedrock and Entra ID for Azure OpenAI, so long-lived keys can be avoided; where manual credentials are needed, they live in Kubernetes Secrets.

Trade-offs. The main deployment target is Kubernetes with Envoy Gateway, so teams without that stack take on both. The documentation reviewed does not describe a vendor telemetry policy, admin audit logs or integration with external secret managers beyond Kubernetes Secrets, and governance is expressed as policy resources rather than budgets per key and team.

Best for: platform teams that already run Envoy Gateway in their clusters and want AI traffic governed through the same declarative resources and GitOps pipeline.

What actually differs between private gateways

Every gateway in this ranking can be self-hosted. The differences that decide an accreditation review are smaller and more specific: which outbound calls happen by default, whether a control plane lives elsewhere, how cloud credentials are handled, and what the auditor receives.

Outbound defaults

The least visible risk in a private LLM stack is a default that reaches the internet. The table lists the ones documented on the pages reviewed for this piece, including the model server most private stacks run behind the gateway.

The vLLM line matters because vLLM is the most common self-hosted model server: its usage stats page states that it “collects anonymous usage data by default” and documents the opt-out variables. Ollama’s documentation notes that it “pulls models from the Internet” and supports HTTPS_PROXY. In a true air gap these calls simply fail, but failing calls produce noisy logs and slow startups, and in an in-VPC deployment they succeed through whatever NAT exists. The safe pattern is to set the flags explicitly and block egress at the network layer as well.

Figure 2: Most outbound calls in a private stack come from defaults, and each one has a documented switch.

Where the control plane lives

Bifrost, LiteLLM and APISIX keep configuration in a database or file inside the deployment. Agent Router keeps it in Kubernetes. Kong offers both models: Konnect hosts the control plane as SaaS with self-hosted data planes, which suits in-VPC deployments that allow an outbound management connection, while self-managed hybrid or traditional mode keeps everything local. Teams should decide this before procurement, because the self-managed route changes which Kong edition and licence they need.

Cloud credentials and private endpoints

For in-VPC teams that still use Bedrock or Azure OpenAI, the strongest pattern combines a private network path with short-lived identity. AWS PrivateLink and Azure Private Link handle the path. Workload identity handles the credentials: Bifrost documents IRSA, EKS Pod Identity, STS AssumeRole, AKS Workload Identity and GKE Workload Identity, and Agent Router documents STS via OIDC and Entra ID. LiteLLM, Kong and APISIX document endpoint overrides that let operators target private hostnames. A gateway that can only take static API keys forces those keys into a secret store and a rotation process, which is workable but adds an item to every audit.

Audit evidence

NIST’s AU control family asks for records of who did what and when, and regulated teams usually need those records in their own SIEM. Bifrost’s enterprise audit logs are HMAC-signed and export as Syslog; Kong’s self-managed audit logs can be RSA-signed and are available through the Admin API; LiteLLM puts audit logs in its Enterprise licence. APISIX and Agent Router rely on general logging and Kubernetes audit trails rather than a gateway-specific admin log. Request logs are a separate question: every gateway here can write them to storage inside the boundary, and teams should confirm that prompt payloads are stored only where the data classification allows.

For a vendor-side view of the same field, Maxim AI’s review of open-source LLM gateways for self-hosted deployments covers overlapping ground and is worth reading alongside this one.

Figure 3: The existing estate and the strictness of the boundary settle most of the choice before features do.

Recommendations keyed to constraints

• You run a fully air-gapped enclave. Bifrost, with the image mirrored and pinned by digest, self-hosted models behind vLLM or SGLang, telemetry scraped by an internal Prometheus, and pricing_url pointed at an internal file. APISIX in standalone mode is the all-open-source alternative if file-based configuration and pattern-matching guardrails are enough.
• You run in a VPC and use Bedrock or Azure OpenAI. Bifrost with workload identity and PrivateLink or Private Link endpoints, so there are no static cloud keys and no public routes. Disable public network access on the Azure resource so the private endpoint is the only path.
• You need FedRAMP- or NIST-aligned evidence. Prioritise signed admin audit logs, an external secret manager and identity federation. Bifrost Enterprise and self-managed Kong Gateway Enterprise both cover all three; LiteLLM Enterprise covers them through its licence.
• You already operate self-managed Kong. Extend it with the AI plugins, set anonymous_reports = off, and budget for the AI licence if you need multi-model load balancing.
• Your platform is Kubernetes and Envoy. Agent Router fits the stack you already run, with OIDC-based cloud credentials, but plan your own audit and secret-management layer.
• Your team is Python-first and wants maximum provider coverage. LiteLLM, with LITELLM_LOCAL_MODEL_COST_MAP set, PostgreSQL and Redis inside the boundary, and an Enterprise licence if secrets must live in Vault.

An offline readiness checklist

Whichever gateway wins, the same steps get it through a security review. They apply equally to the model servers behind it.

1. Mirror and pin. Copy the gateway image, Helm chart and any sidecars into the internal registry, and reference them by digest rather than tag.
2. Inventory egress. List every outbound destination the gateway, the model servers and the databases need, then deny everything else at the network layer.
3. Flip the defaults. Set the offline flags from the table above for every component, including vLLM’s usage statistics.
4. Remove static keys. Use workload identity for cloud providers and a secret manager for anything that remains.
5. Wire the evidence. Send admin audit logs and gateway metrics to internal systems, and confirm retention matches the control framework.
6. Test disconnected. Start the stack with egress fully blocked and confirm it boots, serves a request and logs it without timeouts.

The site’s explainer on what an AI gateway does covers the request path in more detail for readers new to the category.

The verdict

Self-hosting is necessary but not sufficient for a regulated LLM deployment. What separates the five gateways is how much of the private-deployment story each one documents: an offline install path, outbound behaviour that is local or configurable, control planes that stay inside the boundary, workload identity for cloud models, and audit evidence an assessor will accept.

We think Bifrost is the strongest choice for air-gapped and in-VPC deployments in 2026, because its documentation answers each of those questions directly and its open-source core already covers private model routing, local telemetry and budgets, with Vault-backed secrets, signed audit logs and in-VPC support in the enterprise tier. LiteLLM is a capable second for Python-first teams willing to flip its offline flags and license its enterprise features. Kong and APISIX are the right answer where those gateways already run, and Agent Router suits Envoy-native platform teams. Whichever is chosen, the decisive test is the same: block egress, start the stack, and see what breaks.

Feature claims are drawn from public documentation as of October 2026; check current docs before deciding.