What 3 Months of Kubernetes AI Gateway Taught Me About Gateway API's Real Purpose

After writing about my migration from Traefik to Gateway API, a reader emailed me something I couldn’t ignore:

“Great article. But now I need to route LLM traffic through my cluster — GPT-4, Claude, self-hosted vLLM models. Gateway API handles my HTTPRoutes fine, but it has no idea what a ‘token’ is. Do I need a whole separate AI gateway?”

I had the same problem. Our team runs 3 clusters with 47 Gateway resources, and we just added 6 self-hosted LLM services (Llama 3 on vLLM, plus OpenAI and Anthropic via proxy). Within the first week, three things broke:

Rate limiting went useless. Our existing Gateway rate limiter counted requests — but one GPT-4 request can cost 50x more than another. A user sending 100 requests with “Hello” prompts costs less than one user sending 2 requests asking for a code review. Request-based rate limiting is a blunt instrument for LLM traffic.
Round-robin killed GPU utilization. Traditional Kubernetes load balancing sends traffic to pod 1, then 2, then 3 — regardless of which GPU already has the model loaded, which one is half-finished with a batch job, or which LoRA adapter is resident. We were paying $4,200/month for H100 time and getting ~40% utilization.
No model identity. Gateway API routes by path (/api/v1/...) and host (api.example.com). It doesn’t read the request body to see "model": "llama-3-70b". So we couldn’t route gpt-4 to OpenAI and llama-3-70b to our self-hosted vLLM pool without maintaining separate paths — which means every app change requires a Gateway config change.

So I spent 3 months evaluating the emerging Kubernetes AI gateway landscape. What I found surprised me: the Kubernetes community is solving this inside Gateway API itself, not by building a parallel toolchain.

The Kubernetes AI Gateway Working Group

In March 2026, the Kubernetes Steering Committee approved a new working group — WG AI Gateway — specifically to standardize AI workload networking on Kubernetes. This isn’t a vendor project. It’s a CNCF-chartered effort with contributors from Red Hat, Solo.io, Google, Palo Alto Networks, and Bloomberg.

Their charter has four pillars:

Token-based rate limiting — rate limit by input tokens, output tokens, or a weighted cost formula, not by raw request count
Payload inspection — read the LLM request/response bodies for intelligent routing, caching, and guardrails (prompt injection detection, content filtering)
Egress to external AI providers — declarative routing to OpenAI, Anthropic, Bedrock, Vertex AI with managed auth, regional compliance, and automatic failover
Standards-based architecture — layer AI capabilities on top of Gateway API, not replace it

This matters because it means the answer to “do I need a separate AI gateway?” is increasingly: no, your existing Gateway API implementation can become inference-aware.

The Gateway API Inference Extension — Where Routing Gets Smart

The Gateway API Inference Extension is the first concrete output from this effort. It’s an official Kubernetes project (under SIG Network and WG Serving) that transforms any ext-proc-capable gateway — Envoy Gateway, kgateway, GKE Gateway — into an Inference Gateway.

The key insight: instead of round-robin routing, it uses an Endpoint Picker (EPP) that reads real-time server metrics — queue lengths, GPU memory usage, loaded LoRA adapters, prefix cache status — and picks the optimal endpoint per request.

Here’s how it works in practice. You define two new CRDs:

InferencePool (platform admin owns this):

apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: InferencePool
metadata:
  name: llama-3-pool
  namespace: ai-inference
spec:
  targetPortNumber: 8000
  selector:
    app: vllm-llama3
  extension:
    ref:
      name: inference-model-picker

Think of InferencePool as a Service that understands AI workloads. It knows which pods have which models loaded, which GPUs are saturated, and which have prefix caches warmed up.

InferenceModel (ML team owns this):

apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: InferenceModel
metadata:
  name: chat-model
  namespace: ai-inference
spec:
  modelName: llama-3-70b-chat
  targetModels:
    - name: llama-3-pool
      weight: 90
    - name: llama-3-8b-pool
      weight: 10  # fallback to smaller model under load
  criticality: High  # interactive chat — prioritize low latency

The ML team maps public model names to backend pools, sets traffic weights for canary rollouts, and declares criticality. No YAML changes needed when they deploy a new model version — they just update InferenceModel.

The Benchmarks That Convinced Me

The Kubernetes blog published benchmark results on an H100 cluster running 10 Llama 2 replicas with vLLM:

Metric	Standard K8s Service	Inference Extension
Throughput (1000 QPS)	Baseline	Comparable
p90 per-output-token latency (500+ QPS)	High (queue buildup)	Significantly lower
Overall p90 latency (beyond 400 QPS)	Degrades	Reduced tail latency

The throughput stays the same — but tail latency drops dramatically because the EPP prevents the “hotspot” problem where one GPU gets overloaded while another sits idle. For our cluster, that translated to going from ~40% to ~72% GPU utilization. That’s $1,200/month saved on H100 costs without provisioning more hardware.

Envoy AI Gateway — Token-Based Rate Limiting That Actually Works

The Inference Extension handles self-hosted models brilliantly. But what about external providers? This is where Envoy AI Gateway (by Tetrate and Bloomberg, now at v0.4.0) shines.

Envoy AI Gateway provides a unified OpenAI-compatible endpoint that routes to any provider — OpenAI, Anthropic, Bedrock, Vertex AI, or self-hosted — without touching application code. Your apps still call /v1/chat/completions, and the gateway translates to the right provider API.

The killer feature: token-based rate limiting. Here’s the exact configuration we use:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: llm-token-rate-limit
spec:
  targetRefs:
    - name: ai-gateway
      kind: Gateway
      group: gateway.networking.k8s.io
  rateLimit:
    type: Global
    global:
      rules:
        # GPT-4: 1,000 total tokens/hour per user (expensive model)
        - clientSelectors:
            - headers:
                - name: x-user-id
                  type: Distinct
                - name: x-ai-eg-model
                  type: Exact
                  value: gpt-4
          limit:
            requests: 1000
            unit: Hour
          cost:
            request:
              from: Number
              number: 0
            response:
              from: Metadata
              metadata:
                namespace: io.envoy.ai_gateway
                key: llm_total_token

        # GPT-3.5-turbo: 5,000 total tokens/hour per user (cheaper)
        - clientSelectors:
            - headers:
                - name: x-user-id
                  type: Distinct
                - name: x-ai-eg-model
                  type: Exact
                  value: gpt-3.5-turbo
          limit:
            requests: 5000
            unit: Hour
          cost:
            request:
              from: Number
              number: 0
            response:
              from: Metadata
              metadata:
                namespace: io.envoy.ai_gateway
                key: llm_total_token

Envoy AI Gateway automatically extracts token counts from LLM responses in OpenAI schema format. You can also define custom cost formulas using CEL expressions:

llmRequestCosts:
  - metadataKey: custom_cost
    type: CEL
    cel: "input_tokens * 0.5 + output_tokens * 1.5"

We weight output tokens 3x more than input tokens because that’s what drives our actual costs. The previous request-based rate limiter was letting heavy users through and blocking light users — exactly backwards.

agentgateway — Multi-LLM Provider Routing, One Binary

The third piece of the puzzle is agentgateway, which hit v1.0.0 in March 2026 under the Linux Foundation. Where Envoy AI Gateway focuses on provider routing and rate limiting, agentgateway unifies traditional API traffic, LLM routing, MCP (Model Context Protocol), and A2A (agent-to-agent) communication in a single data plane.

Why does this matter? Because before agentgateway, teams running AI agents needed:

An API gateway for their REST services
An AI gateway for LLM routing
A separate proxy for MCP tool servers

That’s three gateways to deploy, monitor, and secure. agentgateway replaces all three with one binary.

Here’s what multi-LLM routing looks like:

llm:
  models:
    - name: claude-haiku
      provider: anthropic
      params:
        model: claude-3-5-haiku-20241022
        apiKey: "$ANTHROPIC_API_KEY"
      matches:
        - headers:
            - name: x-org
              value:
                exact: engineering

    - name: gpt-4
      provider: openAI
      params:
        model: gpt-4o
        apiKey: "$OPENAI_API_KEY"

Route claude-haiku to Anthropic and gpt-4 to OpenAI, with optional header-based routing so the engineering team gets Haiku (fast, cheap) and everyone else gets GPT-4o. All through a single Gateway API listener.

What Went Wrong (Because It Always Does)

1. The ext-proc latency trap

The Inference Extension uses Envoy’s ext-proc (External Processing) filter to route requests through the Endpoint Picker. This adds a network hop. In our first deployment, the EPP was in a different namespace, adding ~3ms of latency per request. For batch jobs, irrelevant. For interactive chat, noticeable.

Fix: Deploy the EPP in the same namespace as your InferencePool, preferably on the same node as your Envoy Gateway pods. The latency dropped to ~200μs — negligible.

2. InferenceModel weight math doesn’t add up to 100

I deployed an InferenceModel with weights 90 + 10 = 100 across two pools. Under load, traffic didn’t split as expected. Turns out, the weight calculation considers available capacity — if the primary pool is saturated, it routes 100% to the fallback regardless of weights.

Fix: This is actually correct behavior. The weights are preferences, not guarantees. The EPP prioritizes latency over weight. If you want strict weight-based splitting, use HTTPRoute-level weight splitting at the Gateway API layer, not the InferenceModel layer.

3. Envoy AI Gateway v0.1 broke on our Azure setup

The v0.1 release had a bug with Azure OpenAI authentication that was patched in v0.2.0. We lost half a day to 401 errors before finding the release notes.

Fix: Don’t run v0.x releases in production without a rollback plan. We now pin to specific versions and test provider connectivity before rolling out. The v0.4.0 release is much more stable — Azure auth, GCP Vertex AI, and Anthropic all work out of the box.

Where This Is Going: The Convergence

Here’s what I see happening in the next 6-12 months:

The Inference Extension goes GA. It’s currently alpha but the Kubernetes blog benchmarks prove the concept. Once it hits beta, every Gateway API implementation will support it — Envoy Gateway, Istio, Cilium, Traefik (which already supports Gateway API v1.5).
Envoy AI Gateway and agentgateway converge. They solve overlapping problems from different angles. Envoy AI Gateway is provider-focused (routing, rate limiting, token tracking). agentgateway is agent-focused (MCP, A2A, unified data plane). Both build on Gateway API. Expect collaboration through the WG.
The “model-as-a-service” pattern becomes standard. Instead of each team managing their own OpenAI keys and provider SDKs, the platform team runs a Gateway AI gateway that handles auth, rate limiting, cost tracking, and provider failover. Developers call one endpoint. This is the same pattern we saw with database connections (one pool, many consumers) and is now happening for AI.

When I’d Still Use a Dedicated AI Gateway

I’m bullish on Gateway API for AI routing, but there are scenarios where a dedicated tool (LiteLLM, OpenRouter) still makes sense:

Scenario	Gateway API + Inference Extension	Dedicated AI Gateway
Self-hosted LLMs on K8s	✅ Native fit	❌ Overkill
Multi-provider routing (OpenAI + Anthropic + Bedrock)	✅ Works well	✅ Also works
Non-Kubernetes infrastructure	❌ Requires K8s	✅ Provider-agnostic
Teams without K8s expertise	❌ Steep learning curve	✅ Managed, simpler
MCP / agent-to-agent protocols	✅ agentgateway	❌ Not supported
Token-based cost control	✅ Built-in	✅ Also built-in

My rule: If you’re already on Kubernetes and running Gateway API, the AI gateway extensions are the natural next step. If you’re not on K8s, a managed AI gateway is the right call. Don’t adopt Kubernetes just for AI routing — but if you have K8s, the AI gateway capabilities are coming to your existing infrastructure.

Common Objections

“Gateway API + AI is too new for production.” The WG launched in March 2026, yes. But the underlying Gateway API is GA (v1.5), and the Inference Extension has been running in GKE and Istio for months. The AI capabilities are extensions — they layer on proven networking primitives, they don’t replace them.

“Why not use LiteLLM or a dedicated AI gateway?” Dedicated AI gateways solve the same problem outside Kubernetes. For K8s-native teams, Gateway API extensions mean no new infrastructure, same RBAC, same observability stack, same operational model. If you already run Envoy Gateway for your HTTP traffic, adding inference routing is a CRD away.

“Token-based rate limiting sounds complex.” It’s actually simpler than request-based once you set it up, because you stop firefighting “why did user X get blocked but user Y didn’t?” The token cost maps directly to your actual bill. When you rate limit by tokens, your budget surprises disappear.

What I’d Do Differently

Start with Envoy AI Gateway first, before the Inference Extension. It’s easier to set up and gives you immediate value (token rate limiting, multi-provider routing). The Inference Extension is more complex and requires GPU infrastructure.
Set up cost tracking from day one. Envoy AI Gateway’s token tracking gives you per-user, per-model cost data. Without it, you’re flying blind on LLM spend. We found one user account responsible for 34% of our monthly API costs — something we only caught because of token-level tracking.
Don’t run v0.x in production without a rollback plan. The AI gateway ecosystem is moving fast (Envoy AI Gateway went from v0.1 to v0.4 in months). Pin versions, test provider connectivity, and keep your old Gateway configs ready to revert.

The Decision Matrix

Your Situation	Recommended Approach
Self-hosted LLMs on Kubernetes	Gateway API Inference Extension + Envoy Gateway
Multi-provider routing (OpenAI, Anthropic, Bedrock)	Envoy AI Gateway
AI agents with MCP / A2A protocols	agentgateway
Already on K8s + Gateway API	Add Inference Extension to existing setup
Not on Kubernetes	LiteLLM, OpenRouter, or managed AI gateway
Small team, one LLM provider	Direct SDK calls — skip the gateway for now

The Bottom Line

Gateway API was designed for HTTP traffic. But the Kubernetes community is extending it — not replacing it — for AI workloads. The Inference Extension makes your gateway model-aware. Envoy AI Gateway adds token-based cost control. agentgateway unifies agent protocols with traditional APIs.

Three months ago, I thought AI routing required a separate toolchain. I was wrong. The best AI gateway is the one you already have — upgraded.

If you’re on Kubernetes and already using Gateway API, the AI extensions are coming to your existing infrastructure. If you’re not on Kubernetes yet, this might be the compelling reason to start.