Best AI Gateways in 2026: 11 Platforms Compared
The AI gateway became a recognized category in 2026 — Gartner published its first Market Guide in February. This guide compares eleven of them, from microsecond-latency proxies to platforms where the gateway is one layer of a governance stack. Ordered by buyer segment, not a fake ranking.
Alejo HernandezCTO, KosmoyAll eleven products here put a policy point between applications and models: one API over many providers, with routing, budgets, guardrails and logs. Beyond that shared core they diverge sharply — lightweight proxies tuned for latency, edge services with free tiers, API-management extensions, a data-platform gateway, and one AI management platform that happens to include a gateway. Gartner's first Market Guide for AI Gateways (February 2026) confirmed the category has hardened into enterprise infrastructure.
This guide does not pretend there is a single winner. Vendors are grouped by the buyer they fit, every competitor claim is cited to the vendor's own material, and the limits are real — including Kosmoy's. If your problem is raw traffic, several products below beat Kosmoy at it; if your problem is proving control over AI, the comparison tilts the other way.
What changed in this update (August 16, 2026): added Databricks Unity AI Gateway (formerly Mosaic AI Gateway), which reached general availability on August 4, 2026 — the strongest platform-native option for Databricks estates. Existing vendors' scores are unchanged.
What counts as AI gateways in 2026
What counts as an AI gateway in 2026: a runtime enforcement point on AI traffic — a unified, usually OpenAI-compatible API over multiple providers; routing with failover; per-team budgets and rate limits; in-path guardrails that can block, not just log; and request-level observability. The 2026 additions are protocol breadth (governing MCP tool calls and agent-to-agent traffic, not just LLM completions) and spend enforcement that stops a request rather than reporting it afterwards.
Buyers confuse the category with four neighbors. LLM observability and eval platforms (LangSmith, Langfuse) watch traffic but do not broker it. AI governance platforms (Credo AI, OneTrust) manage the compliance program of record but mostly have no runtime data path. Hosted model marketplaces blur in from the other side — OpenRouter makes this list because it enforces policy in-path. And AI security platforms are converging on the same chokepoint: Portkey's May 2026 acquisition by Palo Alto Networks folded the category's best-known gateway into Prisma AIRS.
How we scored the field
Every product is scored 0–10 on the same ten capability axes. A 10 is reserved for categorical architectural facts; specialists are expected to outscore platforms on their own spoke, and the scores show it.
AI Inventory & Discovery
Whether the product can answer 'what AI do we run?' beyond its own traffic — registries, agent-harvesting connectors, shadow-AI flagging.
Security & Shadow AI
Credential handling (key vaulting, scoped tokens), RBAC/SSO/SCIM, and detection of AI use that never touches the gateway.
Observability & FinOps
Request logging and tracing, cost attribution per team/key/app, budget alerts, and export to standard telemetry.
Gateway & Policy Control
The core brokerage job: provider breadth, routing/failover, rate limiting, caching, and MCP plus A2A coverage alongside LLM calls.
Guardrails & Runtime Safety
In-path prompt and response checks — PII, prompt injection, content policy — and whether they block synchronously or only log.
Agent Containment
Sandboxed execution, scoped credentials and kill switches; tool ACLs and spend caps are partial credit. Monitoring an agent is not containment.
Compliance & Audit
Tooling for the customer's obligations — EU AI Act, ISO/IEC 42001, NIST AI RMF evidence — scored above the vendor's own SOC 2 certificates.
Testing, Evals & Red-teaming
Testing, evaluation and red-teaming. Native suites score high; playground comparisons and cookbook guides score low.
Agent Building
Whether teams can build and ship agents on the platform itself, versus only governing agents built elsewhere.
Deployment Sovereignty
Where the software runs and what the vendor sees. SaaS-only scores low; air-gap-capable high; 10 is reserved for architectures with no vendor control plane at all.
The field, scored
| Capability (0–10) | Kosmoy | Portkey | LiteLLM | Kong AI Gateway | Databricks Unity AI Gateway | Cloudflare AI Gateway | TrueFoundry | Bifrost (Maxim AI) | agentgateway (Solo.io) | Azure API Management (AI gateway) | OpenRouter |
|---|---|---|---|---|---|---|---|---|---|---|---|
| AI Inventory & Discovery | 9 | 5 | 4 | 5 | 5 | 2 | 6 | 2 | 3 | 5 | 1 |
| Security & Shadow AI | 8 | 4 | 3 | 6 | 5 | 4 | 5 | 5 | 5 | 5 | 3 |
| Observability & FinOps | 7 | 9 | 8 | 7 | 8 | 7 | 8 | 7 | 6 | 7 | 5 |
| Gateway & Policy Control | 8 | 9 | 9 | 9 | 8 | 8 | 9 | 9 | 9 | 8 | 7 |
| Guardrails & Runtime Safety | 8 | 8 | 6 | 8 | 8 | 6 | 7 | 7 | 7 | 6 | 5 |
| Agent Containment | 9 | 4 | 3 | 4 | 5 | 2 | 6 | 2 | 4 | 3 | 2 |
| Compliance & Audit | 9 | 5 | 4 | 3 | 5 | 2 | 5 | 4 | 2 | 3 | 2 |
| Testing, Evals & Red-teaming | 7 | 3 | 1 | 0 | 4 | 3 | 4 | 3 | 0 | 0 | 0 |
| Agent Building | 6 | 2 | 4 | 2 | 2 | 1 | 4 | 1 | 3 | 1 | 0 |
| Deployment Sovereignty | 10 | 9 | 9 | 9 | 3 | 1 | 9 | 9 | 9 | 4 | 2 |
Bold marks the highest score on each row. 10 is reserved for categorical architectural facts; specialists are expected to outscore platforms on their own spoke.
Capability shape, vendor by vendor
Each panel shows one vendor across the same ten axes. Read it as area: a specialist climbs on its own spoke and falls away on the rest; a platform holds the frontier. The dashed outline is Kosmoy for reference.
From comparison to architecture
Route every AI call through one enterprise policy point.
Apply identity, approved routes, guardrails, budgets and audit logging across LLM, MCP tool and A2A agent traffic in your own Kubernetes.
The vendors, by buyer type
No single 1-to-N ranking survives contact with a real shortlist — the right pick depends on who is buying. Each vendor below is labeled with the buyer it fits best.
Kosmoy
AI management platformRegulated enterprises whose gateway must produce governance evidence
A self-hosted control plane for enterprise AI: one inventory, one policy gateway, one audit trail and a containment sandbox for every model, agent and MCP server a company runs.
Kosmoy is the one product here where the gateway is a layer, not the product. One OpenAI-compatible endpoint fronts LLM, MCP and A2A traffic with RBAC, in-path guardrails and budgets (AI Gateway). Around it: four registries, including a master agent registry whose connectors harvest agents from Azure AI Foundry, Bedrock, Vertex, Salesforce and ServiceNow and flag the unmatched as shadow AI; EU AI Act, ISO/IEC 42001 (aligned, not certified) and NIST AI RMF evidence bundles from the same event log; and the Action Capsule, a kernel-enforced sandbox for agents with a live kill switch.
Deployment is the categorical difference: single-tenant in your own Kubernetes, air-gap capable, no vendor control plane. Italy's central bank and banking regulator and Europe's largest defence and aerospace group run it in production; S&P Global initiated analyst coverage in March 2026.
Honest limits: pure gateways beat Kosmoy on raw latency and edge simplicity — Bifrost and Cloudflare exist for good reasons. It shipped an evaluation and red-teaming suite in 2026 (a 7 on evals) though the pure-play specialists still lead on eval-workflow depth, its no-code agent builder is shallower than dedicated builders, and there is no free tier.
Strengths
- Four registries — AI systems, models, MCP servers and a master agent registry that pulls agents from Azure AI Foundry, Bedrock, Vertex, Salesforce and ServiceNow into one list.
- One OpenAI-compatible gateway enforcing guardrails, RBAC, budgets and logging on every LLM, MCP and A2A call.
- Action Capsule: kernel-enforced sandboxing for agents, MCP servers and private models, with per-task credentials and a kill switch.
Limits
- Evaluation and red teaming shipped in mid-2026 — the suite is comprehensive but newer than the pure-play eval platforms, which still lead on experiment tracking, annotation queues and prompt playgrounds.
- The agent builder covers governed internal use cases; dedicated agent-development platforms go deeper.
- No free or self-service tier — procurement runs through an enterprise sales process.
Portkey
AI gateway & LLM-ops control planeThe most complete pure-gateway feature set — now inside Palo Alto Networks
Portkey is an AI gateway and control plane for production AI — one API to 1,600+ models across 45+ providers, with observability, guardrails, prompt management and MCP/agent access control — acquired by Palo Alto Networks in May 2026.
Feature for feature, Portkey remains the deepest pure gateway on this list: 1,600+ models across 45+ providers behind one API, logging with 21+ analytics metrics, per-key budgets, a guardrails ecosystem of 20+ native checks plus partners, and a deployment ladder from MIT-licensed open source to fully air-gapped (Portkey docs). Its MCP Gateway and April 2026 Agent Gateway made it an early mover on agent-traffic governance.
The open question is ownership: Palo Alto Networks closed its acquisition on May 29, 2026, the public docs changelog stops at April 2026, and the last open-source release predates the deal. It also documents no EU AI Act, ISO/IEC 42001 or NIST AI RMF tooling for customers as of July 15, 2026. Our Kosmoy vs Portkey comparison goes axis by axis.
Strengths
- Category-leading gateway breadth: one OpenAI-compatible API to 250+ LLMs and 1,600+ models across 45+ providers, with retries, fallbacks, load balancing and caching (gateway repo, MIT, ~12.4k stars).
- Deep LLM observability and FinOps: full request logging, 21+ metrics, per-key budgets and rate limits, OpenTelemetry and data-lake export (observability docs).
- A sovereignty ladder rare among gateways: open-source self-host, hybrid VPC data plane, and a documented fully air-gapped enterprise deployment (self-hosting docs).
Limits
- Does not document EU AI Act, ISO/IEC 42001 or NIST AI RMF evidence generation or AI risk classification as of July 15, 2026 (checked the enterprise-offering and feature-comparison docs).
- Inventory and governance cover only assets routed through the gateway — no discovery of AI systems or agents outside it.
- No first-class evals or red-teaming product, and no agent sandboxing or kill switch documented.
LiteLLM
Open-source LLM proxy & AI gatewayOSS-first platform teams standardizing LLM access
LiteLLM is BerriAI's open-source proxy and Python SDK that puts 100+ LLM providers behind one OpenAI-compatible API — with spend tracking, guardrails, MCP and A2A gateways, and an enterprise license that adds SSO, RBAC and audit logs on the same self-hosted deployment.
LiteLLM is the open-source default: MIT core, roughly 53,600 GitHub stars, 100+ providers, weekly releases, and the most mature MCP-gateway story in open source — server registry, access groups, OAuth 2.0 including On-Behalf-Of (GitHub). Budgets attach at every level from organization to key, and the enterprise tier is a license key on your own deployment: no data leaves your environment, air-gapped operation included. A Rust migration announced in June 2026 targets sub-1ms overhead by December.
The trade-offs: you run the infrastructure — Postgres, Redis, upgrades, on-call; enterprise pricing is quote-only; and there is no eval tooling, no compliance evidence, and no visibility into AI that bypasses the proxy.
Strengths
- The de-facto standard open-source LLM gateway: ~53.6k GitHub stars, ~9.8k forks and weekly stable releases, fronting 100+ providers behind one OpenAI-compatible API (BerriAI/litellm).
- Deep FinOps for LLM traffic: budgets and spend attribution per organization, team, project, key and tag, soft-budget alerts, Prometheus and OpenTelemetry metrics, and tool-call tracing since Logs v2 (January 2026) (enterprise docs).
- The most mature MCP-gateway story among OSS gateways: an MCP server registry with per-server access groups, OAuth 2.0 including On-Behalf-Of, and controls over which MCP servers are exposed to the public internet (MCP deployment docs).
Limits
- Does not document EU AI Act, ISO/IEC 42001 or NIST AI RMF mapping, risk classification or compliance-evidence packs as of July 15, 2026; audit capability is gateway admin-action logs plus log export.
- No pre-deployment testing, evaluation or red-teaming offering for customer models and applications.
- No shadow-AI discovery: it cannot see or inventory AI usage that does not flow through the proxy.
Kong AI Gateway
AI gateway on the Kong API platformAPI-platform estates extending proven gateway ops to AI traffic
Kong AI Gateway is the AI extension of Kong's API gateway: a plugin-based data path that proxies, secures, rate-limits, caches and observes LLM, MCP and agent-to-agent traffic — self-managed or via the Konnect SaaS control plane.
Kong's pitch is continuity: the plugins, consumer ACLs and analytics your API team already operates, extended to AI. AI Gateway 3.14 (April 2026) made it the first mainstream gateway to cover LLM, MCP and A2A traffic in one runtime (release blog), with MCP Tool ACLs, token-aware rate limiting and semantic caching on a battle-tested Apache-2.0 core.
The caveats are commercial and scope-related: most AI security and analytics plugins require Enterprise or Konnect tiers, the MCP Registry is a tech preview, and compliance support is positioning rather than tooling. No evals, no agent sandboxing, no org-wide AI inventory.
Strengths
- Coverage of all three AI traffic patterns in one runtime — LLM, MCP and agent-to-agent (A2A) — since Agent Gateway went GA in AI Gateway 3.14 (April 2026); Kong calls it 'the most comprehensive AI gateway for the agentic era'.
- A mature, battle-tested open-source core: Kong/kong is Apache-2.0 with ~43.8k GitHub stars, active development, and DB-less, Kubernetes and hybrid deployment modes hardened on general API traffic.
- A deep MCP governance stack: per-tool ACLs via the ai-mcp-oauth2 plugin (3.13, January 2026), OAuth2 scope-based tool filtering, RFC 8693 token exchange, and an MCP Registry in tech preview (February 2026).
Limits
- Most AI security and analytics capabilities — semantic prompt guard, PII sanitizer, content-safety integrations, advanced token rate limiting, LLM usage analytics — require Enterprise or Konnect tiers; the free OSS tier covers mainly ai-proxy basics.
- Does not document EU AI Act, ISO/IEC 42001 or NIST AI RMF evidence generation, risk classification or governance workflows as of July 15, 2026 — its EU AI Act content is positioning, backed by audit logs.
- No pre-deployment evaluation, red-teaming or model-validation tooling documented.
Databricks Unity AI Gateway
Enterprise AI gateway and runtime AI governance layer built on Unity Catalog (formerly Mosaic AI Gateway)Databricks estates that want governance to arrive with the platform
Databricks' runtime governance layer, built on Unity Catalog, that routes, secures, meters and controls traffic to Databricks-hosted and external models, agents, MCP servers and tools — with budgets, guardrails, service policies and centralized usage and cost tracking. GA since August 2026.
Databricks Unity AI Gateway — formerly Mosaic AI Gateway, GA August 4, 2026 — is what a gateway looks like when a data platform builds it: budgets that alert or hard-block, per-principal rate limits, fallbacks and traffic splitting free of charge, service policies bundling PII redaction and content filters before routing, and MCP servers registered as Unity Catalog securables with tool-level permissions and central audit (docs; GA blog). Cost lands in queryable system tables with per-request tags — the strongest finance-grade spend surface in this cohort — and Contextual Service Policies (Beta) can allow, deny or require approval for agent actions.
The boundary is the workspace: governance is opt-in per application (repoint the base URL), there is no self-hosted or air-gapped deployment — Databricks-managed only — payload logging and usage tracking bill DBUs, several headline features (service policies, agent services, Smart Routing) were still Beta at GA, and it documents no EU AI Act, ISO/IEC 42001 or NIST AI RMF evidence tooling and no sandbox or kill switch as of August 16, 2026.
Strengths
- Budgets with hard spend caps: hit a threshold and the gateway can alert — or block further requests until the budget resets or an admin raises it, with per-user and per-budget spend views (budgets docs; AI spend controls blog).
- Deep cost observability via system tables: per-request usage with custom tags, hourly external-model spend by provider service, target model and requesting principal, and enriched billing records for attribution across users, teams, applications, models and providers (cost observability docs).
- Multi-provider routing with reliability controls: fronts OpenAI (incl. Azure OpenAI), Anthropic, Cohere, Amazon Bedrock, Google Vertex AI and any OpenAI-compatible provider, with automatic fallbacks, traffic splitting and Smart Routing (Beta) by quality, cost, performance and budget (external models docs; GA blog).
Limits
- Governance requires traffic to opt in: apps and coding agents outside Databricks are governed only if they repoint their base URL at the workspace gateway — no passive interception or governance of AI traffic that bypasses it is documented as of August 16, 2026.
- Does not document EU AI Act, ISO/IEC 42001 or NIST AI RMF evidence generation, framework mapping or use-case risk-classification workflows as of August 16, 2026 — compliance support is generic audit/system tables, not framework-specific.
- Does not document agent sandboxing or a kill switch inside the gateway as of August 16, 2026: sandboxed execution is a separate product (Databricks Sandbox), and containment is approximated via permissions, hard budget blocks and contextual service policies.
Cloudflare AI Gateway
Edge AI gatewayThe free entry point, at edge scale
Cloudflare AI Gateway is an edge proxy between applications and major AI providers, adding caching, rate limiting, spend limits, logging and analytics, dynamic routing with fallbacks and retries, stored provider keys and Llama-Guard-based guardrails on Cloudflare's global network.
Cloudflare AI Gateway has the lowest barrier to entry in the category: caching, rate limiting, fallbacks, dynamic routing and analytics are free on all plans, and the May 2026 unified REST API fronts OpenAI, Anthropic, Google and Workers AI through one endpoint (changelog). June 2026 spend limits added enforced dollar budgets that block requests.
The architecture is also the constraint: SaaS-only on Cloudflare's edge, with every prompt transiting Cloudflare's network — documented as incompatible with Cloudflare's own Regional Services and Geo Key Manager. Guardrails rely on a single moderation model billed as inference, and evaluations cover only cost, speed and thumbs-up feedback. A developer tool, not a control plane.
Strengths
- The core gateway — caching, rate limiting, fallbacks, automatic retries, dynamic routing, analytics, basic logging — is free on all plans, the lowest adoption barrier in the category (pricing docs).
- Runs on Cloudflare's global edge with battle-tested scale — Cloudflare has publicly documented scaling AI Gateway to billions of logs (Cloudflare blog).
- One unified REST API to major providers (May 2026) plus dashboard-configured dynamic routing — percentage traffic splits and if/else rules — and fallback chains without client code changes (changelog).
Limits
- No deployment sovereignty: SaaS-only, all prompt and response traffic transits Cloudflare's edge. Cloudflare's own Data Localization Suite docs list AI Gateway as incompatible with Regional Services and Geo Key Manager; Customer Metadata Boundary covers log metadata only.
- No AI governance layer: does not document an org-wide AI inventory, EU AI Act / ISO 42001 / NIST AI RMF tooling, risk classification or agent containment as of July 15, 2026.
- Evaluations are shallow relative to eval-focused platforms: cost, speed and manual human thumbs-up feedback over logged traffic only.
TrueFoundry
Enterprise AI gateway & Kubernetes-native ML platformKubernetes estates that want a gateway and an ML platform together
TrueFoundry is a Kubernetes-native enterprise AI platform that combines LLM, MCP and Agent gateways with model serving, fine-tuning and GPU orchestration — deployable as SaaS, hybrid, self-hosted or fully air-gapped.
TrueFoundry pairs LLM, MCP and Agent gateways with a Kubernetes-native ML platform — model serving, fine-tuning, fractional GPUs — and was named a Representative Vendor in Gartner's Market Guide for AI Gateways in February 2026 (press release). Its air-gapped install guide is among the most explicit in the category, and the June 2026 Seldon AI acquisition adds predictive-ML heritage.
Gaps: no EU AI Act, ISO/IEC 42001 or NIST AI RMF evidence tooling documented as of July 15, 2026; no eval or red-teaming suite; shadow-AI discovery beyond the gateway is nascent (the aitori endpoint agent shipped at v0.1.0 in June 2026).
Strengths
- Full-stack gateway coverage — LLM, MCP and Agent gateways under one control plane — recognized as a Representative Vendor in the Gartner Market Guide for AI Gateways (February 2026).
- A documented air-gapped Kubernetes deployment: all images and Helm charts mirrored to a customer-controlled OCI registry, no outbound network dependencies, local IdP and SIEM (air-gap docs).
- Deep MCP governance: a central registry of approved MCP servers, Virtual MCP Servers that expose only curated tool subsets, per-user identity passthrough and OAuth token management (MCP access control).
Limits
- Does not document EU AI Act, ISO/IEC 42001 or NIST AI RMF mapping, evidence packs or AI-governance reporting as of July 15, 2026 — its compliance posture is SOC 2 Type 2, HIPAA and GDPR.
- No dedicated evaluation, LLM-testing or red-teaming suite — prompt versioning and A/B experimentation only.
- No native agent builder: the platform deploys and governs agents built elsewhere (LangGraph, CrewAI, AutoGen, custom).
Bifrost (Maxim AI)
Open-source high-performance AI gatewayRaw throughput on self-hosted infrastructure
Bifrost is Maxim AI's Go-based open-source gateway — vendor benchmarks claim ~11 microseconds of overhead at 5,000 RPS — unifying 1,000+ models across 23+ providers behind one OpenAI-compatible API, with virtual keys and budgets, enterprise guardrails, an MCP gateway and clustering.
Bifrost, the Go-based Apache-2.0 gateway from Maxim AI, is the performance play: Maxim reports 11 microseconds of added latency at 5,000 RPS — a vendor-run benchmark, not independently verified (GitHub). The OSS core covers 23+ providers with failover, semantic caching and MCP support; the enterprise edition adds clustering, guardrail chaining across Bedrock, Azure Content Safety, GraySwan and Patronus, and documented air-gapped installs with no telemetry.
It is deliberately narrow: no org-wide inventory, no agent containment, no compliance tooling, and evals live in the separate Maxim platform. Guardrails and clustering sit behind the enterprise tier, so the free core is a fast proxy rather than a governed gateway.
Strengths
- Performance is the design center: vendor-run benchmarks report ~11 microseconds of added latency at 5,000 RPS, marketed as "50x faster than LiteLLM" (Bifrost repo).
- Fully open-source core (Apache-2.0) covering 23+ providers and 1,000+ models with failover, weighted load balancing, semantic caching and MCP support included.
- Documented air-gapped, on-prem and in-VPC enterprise deployment with no telemetry or phone-home — a strong sovereignty story for regulated buyers (enterprise deployment docs).
Limits
- No org-wide AI inventory or shadow-AI discovery — visibility is limited to traffic that flows through the gateway (not documented as of July 15, 2026).
- No built-in evals or red-teaming in the gateway; those require the separate Maxim AI platform.
- No agent containment — sandboxing and kill-switch features are not documented as of July 15, 2026.
agentgateway (Solo.io)
Cloud-native AI/agent gateway (Envoy + Rust data planes)Cloud-native and Envoy shops governing agent traffic
agentgateway — the current name for what Solo.io formerly marketed as Gloo AI Gateway — is a Linux Foundation-governed, Rust-based open-source data plane for LLM, MCP and agent-to-agent traffic, sold commercially as Gloo Gateway 2.0 and Solo Enterprise for agentgateway.
Solo.io's agentgateway (formerly Gloo AI Gateway) is the infrastructure purist's option: a Rust data plane purpose-built for agent traffic — native MCP and A2A, tool federation, CEL-based RBAC — donated to the Linux Foundation in August 2025 with contributions from AWS, Microsoft, Red Hat, IBM and Cisco (Linux Foundation). The OSS end of this category is now genuinely foundation-governed: alongside it, Envoy AI Gateway reached v1.0 under the CNCF in June 2026.
Caveats: the naming is in flux (Gloo AI Gateway → agentgateway → Solo Enterprise), enterprise pricing is quote-only, air-gap support is undocumented, and there is no compliance, eval or inventory layer — this is a data plane, not a governance product.
Strengths
- Neutral open governance: agentgateway was donated to the Linux Foundation in August 2025, with contributions from AWS, Microsoft, Red Hat, IBM and Cisco (Linux Foundation press release).
- A purpose-built Rust data plane with native MCP and A2A protocol support — tool federation, OpenAPI-to-MCP conversion, OAuth for tools — deeper agent-protocol coverage than generic API gateways (agentgateway repo).
- Mature Envoy / Kubernetes Gateway API pedigree via kgateway (CNCF), with prompt guards, semantic caching, token-based rate limiting and model failover from the Gloo AI Gateway line.
Limits
- An infrastructure-layer product: no org-wide AI inventory, EU AI Act / ISO 42001 compliance tooling, evals or FinOps chargeback documented as of July 15, 2026.
- Product naming is in flux — Gloo AI Gateway is being folded into agentgateway / Gloo Gateway 2.0 / Solo Enterprise for agentgateway — which complicates evaluation and procurement.
- No public pricing; the enterprise editions require a sales engagement.
Azure API Management (AI gateway)
Cloud API management platform with AI gateway capabilitiesAzure-committed estates governing AI inside APIM
Microsoft's "AI gateway capabilities in Azure API Management" are not a separate product but a policy set inside APIM for securing, scaling and observing LLM APIs, MCP servers and A2A agent APIs — with token limits, semantic caching, load balancing and Microsoft Foundry integration.
For a shop standardized on Azure OpenAI and Microsoft Foundry, the AI gateway capabilities in Azure API Management are the path of least resistance: token quotas per subscription or custom key, semantic caching, priority load balancing that spills from PTU to pay-as-you-go, circuit breakers, and a preview unified model API from Build 2026 translating one endpoint to Anthropic, Vertex and Bedrock backends (Microsoft docs). Microsoft cites 1,200+ enterprise customers.
It is not a neutral platform: you run and pay for APIM in Azure, capabilities vary by tier, policy is XML, guardrails are limited to the Azure AI Content Safety integration, and the control plane stays in Azure. AI-specific compliance, evals and containment live in other Microsoft products or nowhere.
Strengths
- Deep first-party Azure integration: managed identity to Azure AI services, Azure Monitor / Application Insights telemetry and Foundry portal integration (AI gateway docs).
- Granular token economics: the llm-token-limit policy enforces TPM limits and hourly-to-yearly token quotas per subscription key, IP or custom key, with prompt-token pre-calculation that rejects over-limit prompts before they reach the backend.
- Production resiliency built in: round-robin, weighted, priority-based and session-aware load balancing (including PTU-first spill-over) plus circuit breakers with Retry-After-aware recovery, and Redis-backed semantic caching.
Limits
- Azure-anchored: requires running and paying for an APIM instance, capability availability varies by tier, and it is not a neutral multi-cloud product.
- Guardrails run through Azure AI Content Safety integration only — no native PII-redaction policy or pluggable third-party guardrail ecosystem as of July 15, 2026.
- No AI-specific compliance tooling (EU AI Act, ISO 42001), no evals, and no agent sandboxing — those live in other Microsoft products (Purview, Foundry) or third parties.
OpenRouter
Multi-model API marketplace / hosted LLM routerDevelopers who want every model behind one hosted API
OpenRouter is a hosted API marketplace exposing 400+ models from dozens of providers behind one OpenAI-compatible endpoint, with provider routing and failover, data-policy controls (ZDR, no-training routing) and, since 2026, organization-level Workspaces and Guardrails.
OpenRouter is the marketplace end of the spectrum: 400+ models behind one OpenAI-compatible endpoint with pass-through provider pricing, automatic failover, and unusually strong data-policy routing — per-request Zero Data Retention enforcement and no-training provider exclusion. Its 2026 enterprise push added Organizations, Workspaces and Guardrails, and the company reported 25 trillion tokens per week at its $113M Series B in May 2026 (Business Wire).
It is SaaS-only — every request transits OpenRouter's cloud, with EU in-region routing the only residency lever — and platform fees apply on credits and BYOK volume. No inventory, compliance, eval or containment story: a superb access layer, not a governance one.
Strengths
- The largest neutral multi-model marketplace: 400+ models behind one OpenAI-compatible API with pass-through provider pricing (OpenRouter FAQ).
- Proven scale and momentum: OpenRouter reported 25 trillion tokens per week at its $113M CapitalG-led Series B in May 2026, at a roughly $1.3B valuation (Business Wire).
- Strong data-policy routing: per-request controls to exclude providers that store or train on data, Zero-Data-Retention-only routing, and no prompt retention by default.
Limits
- SaaS-only: no self-hosted, VPC or air-gapped deployment — all traffic transits OpenRouter's cloud, with EU in-region routing as the only residency mitigation.
- Not a governance platform: no AI inventory, no EU AI Act / ISO 42001 tooling, no evals or red-teaming, and no agent building or containment documented as of July 15, 2026.
- Platform fees on credits and BYOK usage sit on top of provider prices, which matters at high volume (pricing).
Questions buyers ask
Which AI gateway is fastest?
On vendor-reported numbers, Bifrost — Maxim claims 11 microseconds of added latency at 5,000 RPS, though the benchmarks are its own. Portkey's open-source README claims sub-1ms latency, LiteLLM's Rust migration targets sub-1ms overhead by December 2026, and Cloudflare wins on proximity at the edge. Kosmoy does not compete on raw latency: its value is what happens around the call — inventory, evidence, containment — not shaving microseconds off it.
Is an open-source AI gateway enough for an enterprise?
For the traffic problem, often yes: LiteLLM, Kong's OSS core, Bifrost and the foundation-governed data planes (agentgateway, Envoy AI Gateway v1.0) are production-grade, and enterprise tiers add SSO, audit logs and SLAs. What no OSS gateway provides as of July 15, 2026 is the governance layer — organization-wide AI inventory, EU AI Act or ISO/IEC 42001 evidence, agent sandboxing. Enterprises that need those assemble multiple tools or adopt a platform that includes the gateway.
Do I need an AI gateway if I already use Azure OpenAI or Bedrock?
If your estate is genuinely single-cloud, the hyperscaler option is strong — Azure API Management covers quotas, caching and failover for Azure estates. The case for an independent gateway appears when reality is multi-provider: teams on different clouds, an acquisition that brings a second stack, or a sovereignty requirement the hyperscaler cannot meet. An OpenAI-compatible gateway keeps the switching cost to a base-URL change.
What does an AI gateway cost?
The widest range in this stack: free (Cloudflare's core features, every OSS self-host), self-serve tiers ([Portkey](https://portkey.ai/pricing), [TrueFoundry](https://www.truefoundry.com/pricing)), usage fees (OpenRouter), and enterprise quotes (Kong, LiteLLM, Bifrost, [Kosmoy](https://www.kosmoy.com/pricing/)). Two costs hide off the price list: operating a self-hosted deployment yourself, and integrating the governance tools a pure gateway lacks.
Which AI gateway is best for EU AI Act compliance?
None of the pure gateways document EU AI Act, ISO/IEC 42001 or NIST AI RMF tooling as of July 15, 2026 — their compliance stories are their own SOC 2 certificates. Kosmoy is the exception on this list: it classifies systems by EU AI Act risk tier and generates framework-mapped evidence from gateway logs. Note the timeline: after the May 2026 Digital Omnibus, high-risk obligations land in December 2027 and August 2028, but Article 50 transparency obligations still applied from August 2, 2026.
Can I run two gateways at once?
Yes, and many enterprises do — everything here speaks OpenAI-compatible APIs, so gateways chain or serve different populations. A common pattern pairs a developer-facing gateway (LiteLLM, Portkey, Cloudflare) with a governed path to production ([Kosmoy's LLM Gateway](/platform/llm-gateway/)) that holds the inventory, compliance evidence and agent containment. Migration in either direction is configuration, not a rewrite.
Methodology
Each vendor was scored 0–10 on the ten axes above from a dossier of its own documentation, changelogs, repositories and press, verified as of July 15, 2026. Competitor performance and scale numbers are reported as the vendor's claims with citations, never as our measurements. A 10 is reserved for categorical architectural facts, any score of 7 or higher must be defensible from cited evidence, and gaps are phrased 'does not document X as of July 15, 2026'.
On the roster: we excluded Helicone, whose gateway would otherwise have earned a place — it has been in maintenance mode since the March 2026 Mintlify acquisition, fine for existing users but a risk for new adoption (Helicone announcement). Gartner's February 2026 Market Guide frames the same category boundary; we treat it as validation, not as a ranking source.
Disclosure: Kosmoy publishes this guide. The mitigations are structural — every competitor claim is cited to that competitor's material, every entry carries its limits (including ours), and the verdict names competitors for three of the four buyer types. If you are weighing build vs buy for the gateway layer, the same evidence standard applies.
Sources
Every factual claim about another vendor on this page traces to that vendor's own published material or a named third-party source below.
- Kosmoy AI Gateway — accessed July 15, 2026
- TrueFoundry recognized as a Representative Vendor in the Gartner Market Guide for AI Gateways (Business Wire, Feb 20, 2026) — accessed July 15, 2026
- Helicone — Joining Mintlify (March 2026) — accessed July 15, 2026
- Palo Alto Networks press release — Portkey acquisition completed (May 29, 2026) — accessed July 15, 2026
- LiteLLM Rust migration announcement (GitHub issue #31263, June 25, 2026) — accessed July 15, 2026
- Kong AI Gateway 3.14 release (Agent Gateway GA, April 2026) — accessed July 15, 2026
- OpenRouter $113M Series B, 25T tokens/week (Business Wire, May 26, 2026) — accessed July 15, 2026
- Kosmoy Platform — accessed July 15, 2026
- Kosmoy Action Capsule — accessed July 15, 2026
- Kosmoy AI Compliance — accessed July 15, 2026
- Kosmoy AI Evaluation & Red Teaming — accessed July 31, 2026
- Portkey open-source gateway repository — accessed July 15, 2026
- Portkey docs — what is Portkey — accessed July 15, 2026
- Portkey docs — plan & feature comparison (SaaS / hybrid / air-gapped) — accessed July 15, 2026
- Portkey docs — observability — accessed July 15, 2026
- Portkey docs — guardrails — accessed July 15, 2026
- Portkey docs — MCP gateway — accessed July 15, 2026
- Portkey pricing — accessed July 15, 2026
- LiteLLM GitHub repository (stars, license, activity) — accessed July 15, 2026
- LiteLLM README (100+ providers, MCP/A2A, performance claims) — accessed July 15, 2026
- LiteLLM enterprise docs (features, SLAs, air-gap, pricing by quote) — accessed July 15, 2026
- LiteLLM release notes index (2026 releases) — accessed July 15, 2026
- Guardrail policy templates (incl. offline/air-gapped mode) — accessed July 15, 2026
- MCP deployment docs (registry, exposure controls, air-gap guidance) — accessed July 15, 2026
- litellm-agent-runtime (per-session VM coding-agent runtime) — accessed July 15, 2026
- Kong AI Gateway product page — accessed July 15, 2026
- A2A support press release (PR Newswire, April 2026) — accessed July 15, 2026
- MCP Tool ACLs announcement (AI Gateway 3.13, January 2026) — accessed July 15, 2026
- Kong MCP Registry press release (February 2026) — accessed July 15, 2026
- Konnect LLM usage reporting docs — accessed July 15, 2026
- Kong EU AI Act positioning blog — accessed July 15, 2026
- Kong/kong GitHub repository — accessed July 15, 2026
- Kong pricing — accessed July 15, 2026
- AI governance with Unity AI Gateway (Databricks docs) — accessed August 16, 2026
- Unity AI Gateway product page — accessed August 16, 2026
- Unity AI Gateway is Generally Available (Databricks blog, Aug 4, 2026) — accessed August 16, 2026
- Manage budgets for Unity AI Gateway — accessed August 16, 2026
- Monitor Unity AI Gateway cost — accessed August 16, 2026
- External models in Model Serving (provider list) — accessed August 16, 2026
- What's new: service policies, guardrails, observability and cost controls — accessed August 16, 2026
- Unity AI Gateway pricing — accessed August 16, 2026
- Mosaic AI Gateway updates (former product name; Databricks blog) — accessed August 16, 2026
- Cloudflare AI Gateway docs — accessed July 15, 2026
- AI Gateway pricing — accessed July 15, 2026
- Guardrails feature docs — accessed July 15, 2026
- Changelog — spend limits (June 5, 2026) — accessed July 15, 2026
- Changelog — unified REST API (May 21, 2026) — accessed July 15, 2026
- Data Localization Suite docs (AI Gateway compatibility) — accessed July 15, 2026
- Blog — scaling AI Gateway to billions of logs — accessed July 15, 2026
- Blog — AI Security for Apps GA — accessed July 15, 2026
- TrueFoundry air-gapped deployment docs — accessed July 15, 2026
- TrueFoundry AI Gateway product page — accessed July 15, 2026
- TrueFoundry guardrails overview — accessed July 15, 2026
- Agent Gateway launch press release (Businesswire, June 2, 2026) — accessed July 15, 2026
- TrueFailover launch (VentureBeat, January 2026) — accessed July 15, 2026
- Seldon AI acquisition (SiliconANGLE, June 25, 2026) — accessed July 15, 2026
- aitori repository (v0.1.0, June 25, 2026, Apache-2.0) — accessed July 15, 2026
- Enterprise MCP access control blog — accessed July 15, 2026
- Bifrost GitHub repository — accessed July 15, 2026
- Bifrost product page — accessed July 15, 2026
- Bifrost pricing — accessed July 15, 2026
- Bifrost enterprise deployment (in-VPC, air-gapped) — accessed July 15, 2026
- Bifrost guardrails docs — accessed July 15, 2026
- Guardrails at the Gateway (Bifrost blog) — accessed July 15, 2026
- agentgateway GitHub repository — accessed July 15, 2026
- kgateway GitHub repository (CNCF) — accessed July 15, 2026
- Linux Foundation welcomes agentgateway (Aug 2025) — accessed July 15, 2026
- Solo.io — Introducing Gloo Gateway 2.0 — accessed July 15, 2026
- Gloo AI Gateway docs overview — accessed July 15, 2026
- TechCrunch — Solo.io $135M at $1B valuation (2021) — accessed July 15, 2026
- AI gateway capabilities in Azure API Management (official docs) — accessed July 15, 2026
- llm-token-limit policy — accessed July 15, 2026
- llm-content-safety policy — accessed July 15, 2026
- AI Gateway in APIM available in Microsoft Foundry (preview) — accessed July 15, 2026
- What's new in Azure API Management at Microsoft Build 2026 — accessed July 15, 2026
- Azure API Management pricing — accessed July 15, 2026
- OpenRouter FAQ (fees, data retention) — accessed July 15, 2026
- Provider routing docs — accessed July 15, 2026
- Guardrails announcement — accessed July 15, 2026
- Introducing Workspaces — accessed July 15, 2026
- TechCrunch — OpenRouter valuation to $1.3B — accessed July 15, 2026
Shortlisting for a regulated environment?
Kosmoy puts an inventory, a policy gateway and a containment sandbox around every AI your teams run — in your own Kubernetes.
Or email sales@kosmoy.com.
Where to next
Explore the AI Gateway
Apply identity, approved routes, guardrails, budgets and audit logging across LLM, MCP tool and A2A agent traffic in your own Kubernetes.
Kosmoy vs Databricks Unity AI Gateway: AI Governance and Control Compared (2026)
The direct comparison, scored on the same ten axes.
How Kosmoy is priced
See the deployment, module, workload and support factors that determine enterprise scope.