Inference Gateway vs. AI Gateway: two control planes, one production path

The terms AI gateway and inference gateway are often used as if they describe competing products. They do not. They solve different control-plane problems and can sit in the same request path.

An AI gateway governs how applications consume models: identity, policy, provider abstraction, quotas, safety, routing, usage attribution, and resilience across model endpoints. An inference gateway governs how a self-hosted model request lands on the best serving replica: model-aware routing, queue and accelerator awareness, prefix-cache affinity, LoRA adapter placement, rollout traffic, and serving priority.

That distinction changes the architecture question from Which gateway should we buy? to Which decisions belong at each layer, and who will operate them?

Executive summary

AI gateways manage demand across models and providers. They create a governed API surface for application teams and centralize cross-cutting policy.
Inference gateways manage execution across model-serving replicas. They use model-server signals that ordinary HTTP load balancers do not understand.
They are complementary. A mature platform may route a request through an AI gateway first, then an inference gateway when the selected target is a self-hosted model pool.
Ownership is the decisive boundary. The AI gateway usually belongs to an enterprise API or AI platform team; the inference gateway belongs close to the Kubernetes, accelerator, and model-serving platform.

Start with the control decision, not the label

A conventional load balancer sees connections, paths, status codes, and perhaps latency. LLM serving adds state and economics that are hidden behind an otherwise ordinary HTTP request:

the requested model or adapter;
prompt and completion token volume;
queue depth and active sequences;
KV- or prefix-cache locality;
GPU or TPU saturation;
time to first token (TTFT) and time per output token (TPOT);
regional, provider, tenancy, and data-residency constraints;
safety policy and budget ownership.

No single routing decision optimizes all of these. The clean design separates access-plane decisions from serving-plane decisions.

AI gateway: the access and governance plane

The AI gateway answers questions at the application boundary:

Is this caller authenticated and authorized for this model?
Which provider or deployment satisfies policy, geography, cost, and availability requirements?
How many tokens may this tenant consume?
Should the request use a smaller model, a premium model, or a fallback?
Which safety, audit, redaction, and retention rules apply?
How should usage and cost be attributed?
Can an exact or semantic cache safely answer the request?

An AI gateway can be a managed cloud capability, a commercial gateway, an open-source service, or an internally built platform. Managed product is therefore not the definition. The defining characteristic is the governed model-access surface it presents to applications.

Microsoft, for example, describes AI gateway capabilities in API Management as an extension of its existing API gateway—not a separate gateway type—and includes multi-provider model APIs, token quotas, semantic caching, safety, routing, observability, MCP, and A2A governance. AWS describes an AI gateway as a model-abstraction service that centralizes security, routing, control, and observability across foundation-model endpoints.

Inference gateway: the serving optimization plane

The inference gateway answers a narrower but deeper execution question:

Given that this request is going to a self-hosted model, which pool and replica should process it now?

The Kubernetes Gateway API Inference Extension defines an inference gateway as a gateway or load balancer coupled to an Endpoint Picker. The gateway selects an InferencePool; the Endpoint Picker evaluates model-server metrics and capabilities; the selected endpoint receives the request.

Useful serving signals include:

queue length and active requests;
accelerator utilization;
expected TTFT or TPOT;
prefix-cache or KV-cache affinity;
model and LoRA adapter availability;
request priority;
pool health, capacity, and rollout weight.

This is not merely “an API gateway deployed near GPUs.” It is a scheduling and routing layer that understands the behavior of inference runtimes such as vLLM and the cost of moving a request away from warm model state.

The request path when both are present

A production request can pass through both layers without duplicating responsibility:

The application calls one enterprise endpoint. It supplies its workload identity, tenant context, requested capability, and request payload.
The AI gateway applies access policy. It validates identity, enforces token and budget limits, applies safety controls, records attribution, and selects a provider or model deployment.
The selected target determines the next hop. A managed API call goes directly to that provider. A self-hosted target is sent to the inference gateway for the relevant cluster or region.
The inference gateway selects the serving pool. Model name, version, priority, and rollout policy determine the InferencePool or equivalent backend.
The endpoint picker selects a replica. It uses queue, cache, adapter, and accelerator signals to choose an endpoint.
Telemetry returns through both layers. The inference layer reports serving performance; the AI access layer records tenant usage, policy outcomes, provider selection, and cost.

The architectural rule is simple: do not ask the enterprise gateway to make pod-level scheduling decisions, and do not ask the cluster gateway to become the enterprise system of record for identity, budget, and provider policy.

Capability and ownership matrix

Concern	AI gateway	Inference gateway	Primary owner
Application authentication and authorization	Primary	Pass-through or local enforcement	API / AI platform
Provider and model abstraction	Primary	Limited to served models	AI platform
Token quotas and chargeback	Primary	Emits serving metrics	FinOps / AI platform
Safety, moderation, and data policy	Primary	Optional local integration	Security / AI governance
Multi-provider failover	Primary	Not its main purpose	AI platform / SRE
Semantic or response caching	Common capability, policy-sensitive	Prefix-cache locality only	AI platform
Model- and adapter-aware routing	Coarse target selection	Primary	ML platform
Queue-, cache-, and accelerator-aware endpoint selection	No	Primary	ML platform / SRE
Canary rollout between serving pools	Policy or provider level	Primary inside self-hosted serving	ML platform
GPU/TPU utilization optimization	Indirect	Primary	ML infrastructure
TTFT and TPOT optimization	End-to-end SLO	Replica-level action	Shared
Tenant-level audit and cost allocation	Primary	Supplies measurements	AI platform / FinOps

Why ordinary round-robin fails for self-hosted LLMs

HTTP replicas are not interchangeable when inference state is expensive.

A replica with a useful prefix cache may produce the first token faster than an otherwise idle replica. A pod with the requested LoRA adapter already loaded can avoid adapter transfer and activation. Two replicas reporting the same CPU utilization may have very different active sequence counts, queue depth, GPU memory pressure, or expected decode time.

The Kubernetes inference extension addresses this by separating the proxy from an extensible Endpoint Picker. Google’s GKE implementation documents routing signals including KV-cache hits, accelerator utilization, queue length, model identity, serving priority, and predicted latency. It also supports traffic splitting for model rollout and sharing an accelerator across LoRA adapters.

The implication is operational: the routing layer needs reliable model-server telemetry, and stale telemetry is itself a failure mode. A sophisticated picker with delayed metrics can perform worse than a simpler policy with current signals.

AI gateway controls are not free

Centralization creates leverage, but every policy adds latency, state, and failure modes.

Semantic caching

Semantic caching can reduce provider calls, latency, and token cost by reusing an answer for a sufficiently similar prompt. It can also return a response that is stale, unsafe, incorrectly scoped, or semantically close but operationally wrong.

Partition cache keys by tenant, model, prompt version, tool context, safety-policy version, and relevant data boundary. Avoid semantic caching for personalized, regulated, transactional, safety-critical, or rapidly changing responses unless the application has an explicit validation strategy.

Token limits and quotas

Request-per-second controls are insufficient when one request can consume hundreds of times more tokens than another. Token-aware quotas provide a better demand-control mechanism, but distributed gateways introduce counter consistency questions. Decide whether quotas are local to a gateway, regional, or globally authoritative, and document what happens during partition or failover.

Provider abstraction

A unified API can reduce application coupling, but model APIs are not perfectly fungible. Tool calling, structured output, streaming behavior, safety responses, context limits, and provider-specific controls differ. Preserve a capability model instead of promising a lowest-common-denominator interface that silently discards useful behavior.

Reliability boundaries to design explicitly

Fail open or fail closed

Each policy needs a declared failure posture:

Authentication, data-residency, and mandatory safety policy: normally fail closed.
Optional enrichment or non-critical telemetry: may fail open with alerts.
Endpoint picker: may use a known fallback policy, but should not hide prolonged picker failure.
Semantic cache: bypass on failure rather than blocking all inference.

Streaming and backpressure

LLM responses are long-lived streams. Configure idle timeouts, maximum request duration, cancellation propagation, retry rules, and connection limits for streaming rather than ordinary REST traffic. Never retry a partially streamed generation as though it were an idempotent GET.

Retry budgets

Retries can multiply token cost and amplify overload. Retry only when the failure is known to be pre-execution or safely idempotent. Use circuit breakers and bounded failover; record whether another provider or model produced the final answer.

Telemetry cardinality

Model, tenant, route, provider, adapter, prompt version, and error class are useful dimensions. Raw user identifiers and unbounded prompt-derived labels are not. Define a cardinality budget before exporting metrics, and keep sensitive prompt and response data out of general-purpose logs.

Choose the topology from your workload

Use an AI gateway without an inference gateway when

all models are consumed as managed APIs;
governance, quotas, provider resilience, and cost attribution are the main problems;
the organization does not operate model-serving replicas;
provider-native routing is sufficient.

Use an inference gateway without a separate AI gateway when

one platform team serves a controlled set of internal workloads;
models are self-hosted on Kubernetes;
endpoint selection and accelerator efficiency dominate;
existing identity, API management, and observability already meet governance needs.

Use both when

applications consume managed and self-hosted models through one governed surface;
central policy must coexist with cluster-specific serving optimization;
regulated workloads require explicit routing and residency controls;
multiple teams need shared cost attribution while ML infrastructure retains control of GPU scheduling.

Build, buy, or compose

Decision signal	Prefer managed AI gateway	Prefer self-managed AI gateway	Add inference gateway
Fast onboarding and low operational overhead	Strong fit	Possible, but higher ownership	Only for self-hosted serving
Deep enterprise policy integration	Depends on product extensibility	Strong fit	Complementary
Multi-provider portability	Validate feature parity	Strong fit with deliberate adapters	Not sufficient alone
Air-gapped or tightly controlled data plane	Often constrained	Strong fit	Strong fit
Existing Kubernetes and GPU platform	Optional front door	Optional front door	Strong fit
Prefix cache, LoRA, queue and accelerator awareness	Not the core function	Not the core function	Required capability
Small platform team	Strong fit	Risk of overbuilding	Use only when operational value is measurable

The cost comparison must include more than licence fees. Measure engineering ownership, on-call load, policy development, upgrade cadence, observability, high availability, and the opportunity cost of maintaining a control plane.

A rollout sequence that avoids a gateway monolith

Inventory traffic and ownership. Identify applications, tenants, models, providers, regions, data classifications, current limits, and accountable teams.
Define the stable contract. Standardize identity, request metadata, model naming, streaming, errors, usage records, and correlation IDs before selecting products.
Establish baseline SLOs. Capture success rate, p50/p95 TTFT, TPOT, end-to-end latency, token volume, cost per successful task, and fallback rate.
Introduce the AI access plane in observe mode. Mirror or pass traffic without enforcing broad policy; validate attribution and latency overhead.
Enforce one policy at a time. Start with identity and coarse quotas, then add routing, safety, caching, and provider failover with explicit rollback criteria.
Introduce inference-aware routing where evidence supports it. Compare against round-robin using real workload distributions, not synthetic averages alone.
Test degraded modes. Exercise picker failure, stale metrics, provider throttling, regional outage, streaming cancellation, cache outage, and quota-store partition.
Keep layer ownership separate. Publish a responsibility matrix and escalation path so a slow generation does not bounce indefinitely between API and ML platform teams.

Architecture review checklist

Before production approval, answer these questions:

Is each routing decision assigned to exactly one layer?
Can every request be traced from tenant to provider, pool, replica, and cost record?
Are model aliases versioned and promotion-controlled?
Are quotas token-aware, and is their consistency boundary documented?
Are safety and data-residency controls fail-closed where required?
Are streaming timeout, cancellation, and retry semantics tested?
Does self-hosted routing use current queue, cache, adapter, and accelerator signals?
Can the endpoint picker fail without taking down all inference?
Are cache partitions and invalidation rules explicit?
Is metric and log cardinality bounded?
Are provider fallback and model substitution visible to the caller?
Has the team measured gateway overhead against end-to-end SLOs?

The practical conclusion

An AI gateway is the enterprise policy and access plane. An inference gateway is the self-hosted serving optimization plane. Treating them as rivals leads to either an overloaded enterprise gateway that tries to schedule GPUs, or a cluster gateway that accumulates identity, cost, compliance, and provider policy it was never designed to own.

The strongest architecture composes the layers, keeps their decisions explicit, and measures whether each one improves a production SLO.

Sources

CTA

If your AI platform has accumulated overlapping gateways, model proxies, quota services, and cluster routers, KMS ITC can map the decision boundaries and turn them into a deployable target architecture.