An API gateway manages traffic to services you own: auth, routing, rate limits, WAF, observability. An AI gateway sits in front of model providers you rent and adds what an HTTP proxy cannot see — tokens, prompts, streaming deltas, and model choice. Most production stacks need both, at different layers.
They get conflated because both are called gateways and both emit metrics, yet they answer different questions. An API gateway answers is this caller allowed to reach this service, and how fast? An AI gateway answers which model serves this prompt, what did it cost, and what happens when that vendor degrades?
What a traditional API gateway does
An API gateway is a north-south traffic manager for services you operate: requests arrive from outside, and the gateway decides which internal service handles them, under what conditions. Kong, NGINX/OpenResty, Envoy, Traefik, and AWS API Gateway all converge on five jobs.
- Authentication. Terminate TLS, validate JWTs against a JWKS endpoint, introspect OAuth2 tokens, verify mTLS certs, check API keys — so downstream services trust an identity header.
- Routing. L7 dispatch by host, path, or header, weighted splits for canary deploys, timeouts, connection pooling, health checks.
- Rate limiting. Token-bucket or sliding-window counters keyed on consumer, IP, or key — almost always in requests per window, the unit HTTP understands.
- WAF and edge protection. OWASP Core Rule Set evaluation, bot mitigation, IP reputation, request-size caps, DDoS absorption before traffic reaches your fleet.
- Observability. RED metrics per route and consumer, structured access logs, trace-context propagation into your service graph.
The structural fact that matters: an API gateway operates on the HTTP envelope, not the payload. It knows status codes, byte counts, and headers. It does not know that a 200 OK consumed 4,120 prompt tokens, or that the prompt held a customer’s email. That blindness makes a generic gateway fast at the edge — and makes it unable to manage model spend.
One consequence breaks more LLM rollouts than anything else: request-count limits are near-meaningless for LLM traffic. Sixty requests per minute sounds reasonable until one request is a 50-token classification and the next is a 200,000-token document analysis. Same counter, wildly different cost — and your worst-behaved tenant stays invisible until the invoice arrives.
What an AI gateway adds
An AI gateway is an egress-oriented proxy specialized for model APIs. It speaks the provider protocols (OpenAI-compatible chat completions, Anthropic messages, Gemini generateContent) and parses the payload. Six capabilities distinguish it.
1. Multi-provider routing and schema normalization
One request shape, translated to each vendor’s dialect: role naming, tool-call encoding, system-prompt placement, and error taxonomy. When one vendor returns 429 rate_limit_exceeded and another returns 429 overloaded_error, the gateway maps both to one internal class so your application writes one branch — which is what lets you switch AI providers as a config change rather than a refactor.
2. Token metering and budget enforcement
Token counts — prompt, completion, cached — are attributed per request to a key, team, feature, or tenant. That enables pre-flight rejection (estimate against budget, then downshift or refuse before spending) and per-feature attribution: the only way to know whether the summarizer or the chat assistant eats the budget.
3. Prompt-logging control
Because the gateway parses the body, it can enforce what a generic proxy physically cannot: store nothing, metadata only, a sampled percentage, or full text with regex redaction of emails, phone numbers, and credential-shaped strings. Retention windows and per-tenant opt-outs live here — often why an AI gateway clears a security review at all.
4. Capability-aware model fallback
Fallback does not mean retrying the same endpoint. The gateway knows which models are substitutes — same tier, comparable quality, compatible context window, ideally a different vendor and failure domain — and reroutes on 429, 5xx, or timeout. That needs a capability registry, not a retry loop; our guide to multi-provider failover covers the circuit-breaker mechanics.
5. Streaming-aware proxying
Responses arrive as Server-Sent Events, and the failure is subtle: a buffering proxy delivers nothing for twenty seconds, then dumps the whole answer. Users read that as broken. An AI gateway disables buffering on the streaming path, preserves chunk boundaries, forwards the terminal [DONE], counts tokens from the stream, and cancels the upstream call when the client disconnects.
6. Semantic cache
Exact-match HTTP caching is useless here, because two prompts with the same intent are almost never byte-identical. A semantic cache embeds the prompt, finds a stored prompt above a cosine-similarity threshold, and returns the prior completion. Start precision-first around 0.95 and lower it only after measuring false hits.
Three constraints keep it honest: cache only low-temperature tasks; scope entries by model, temperature, and system prompt; and never cache responses derived from permissioned data, since a hit would leak one tenant’s answer to another.
The overlap and the gaps
Plenty of capabilities appear in both columns, which is why teams assume one replaces the other. The gaps are what matter.
| Capability | Traditional API gateway | AI gateway |
|---|---|---|
| Auth (JWT, OAuth2, mTLS, keys) | Yes — mature, standards-based | Partial — a bearer key for internal callers |
| Request-count rate limiting | Yes — the core strength | Yes |
| Token-based quota & spend budget | No — cannot see tokens | Yes — per key, team, tenant |
| Routing by path / host / header / weight | Yes | Limited — routes by model, not by service |
| Multi-vendor schema normalization | No | Yes |
| Capability-aware model fallback | No — retries the same upstream | Yes — crosses vendor boundaries |
| Streaming (SSE) pass-through | Possible, but needs deliberate tuning | Yes — streaming-native |
| Token metering & usage attribution | No | Yes |
| Prompt logging with redaction | No — payload is opaque | Yes — payload-level policy |
| Semantic caching | No | Yes |
| WAF, DDoS, bot mitigation | Yes — why it sits at the edge | Rarely |
The traditional gateway owns who gets in and how much traffic they send; the AI gateway owns what that traffic costs and which model serves it. The two “No” rows in the middle column — token budgets and cross-vendor fallback — are the entire reason AI gateways exist as a category.
Where each sits in the stack
Direction is the cleanest way to remember the boundary. An API gateway handles inbound traffic to services you own. An AI gateway handles outbound traffic to vendors you rent from. They sit at opposite ends of the request path:
Client
|
v
[ Edge API gateway ] TLS, user auth, WAF, per-consumer request limits
| knows nothing about tokens or models
v
[ Your application ] business logic, prompts, tool definitions
| holds an INTERNAL token, never a vendor key
v
[ AI gateway / egress ] model routing, token budgets, vendor fallback,
| semantic cache, prompt-logging policy
v
[ Provider A ] [ Provider B ] [ Provider C ]Three rules make that boundary work.
- User authentication stays at the edge. The AI gateway authenticates your application to the model layer with an internal credential — a credential boundary between you and your vendors, not a user-identity boundary. A leaked internal token cannot reach your services; a compromised user session cannot spend your inference budget.
- Vendor keys never leave the AI gateway. Neither your application code nor your client devices hold a provider key. That is the largest security win of the egress layer, and it makes rotation a one-place operation.
- Correlate traces across both hops. Propagate one trace ID from the edge into the gateway, or you will see p99 spike without knowing whether the cause was your service or a provider’s.
A representative egress config shows how much model-specific behavior collapses into one place:
# AI gateway egress config (gateway-agnostic, illustrative)
server:
direction: egress # outbound to providers only
auth: internal-token # user auth handled at the edge gateway
routes:
- name: chat-completions
match: { path: /v1/chat/completions }
model_tiers: # route by tier, not hard-coded model name
fast: [gpt-mini-class, claude-haiku-class]
mid: [gpt-class, claude-sonnet-class]
strong: [gpt-frontier-class, claude-opus-class]
fallback:
trigger: [429, 500, 502, 503, 504, timeout]
strategy: cross_vendor # never retry the vendor that just failed
max_attempts: 3
total_budget_ms: 25000
streaming:
buffer: false # SSE chunks must pass through untouched
cancel_upstream_on_disconnect: true
metering:
unit: tokens # not requests
emit: [prompt_tokens, completion_tokens, cached_tokens, model, vendor]
budget: { window: 1h, on_exceed: throttle }
cache:
semantic: { enabled: true, similarity: 0.95, ttl_seconds: 3600 }
logging:
store_prompt: sampled # none | metadata_only | sampled | full
sample_rate: 0.05
redact: [email, phone, api_key]Everything in that file is invisible to a generic proxy, and none of it belongs in application code.
Common mistakes
Mistake 1: using Kong, NGINX, or an ALB for LLM routing
These are excellent products doing a different job. Pointed at a model API they fail four ways: request-count limits do not track token cost; SSE breaks unless you disable buffering, and the default is on; vendor error taxonomies flatten, so a 429 retries the same overloaded vendor instead of failing over; and counting tokens means parsing a body that can exceed 100 KB.
# A plain reverse proxy in front of an LLM API — three traps marked.
location /v1/chat/completions {
proxy_pass https://api.provider.example;
proxy_set_header Authorization "Bearer $UPSTREAM_KEY";
proxy_buffering off; # TRAP 1: default is "on". SSE chunks are held
# until the response completes, so the UI shows
# nothing for 20s then the whole answer at once.
proxy_read_timeout 300s; # TRAP 2: LLM calls exceed the 60s default.
limit_req zone=llm burst=5; # TRAP 3: counts REQUESTS, not tokens. A tenant
# sending 200k-token prompts costs 1000x another
# and looks identical in this counter.
}You can close these gaps with custom plugins — but you are then maintaining a model-routing layer inside a web server, with no capability registry, no token accounting, and no semantic cache.
Mistake 2: using an AI gateway as your only auth layer
AI gateways ship pragmatic auth — usually a bearer key per caller — because their job is to identify a budget, not a user. That is a different question from “may this person read invoice 4471?” They have no WAF, no bot mitigation, no OAuth2 scope model, and no tenant RBAC over your resources. Exposing one directly to browsers puts a token-billing endpoint on the public internet with no edge protection.
Mistake 3: no per-request cost attribution
If token usage lives only on provider dashboards, you have N dashboards and no way to answer “which feature got expensive last Tuesday.” Meter at the gateway, where request, caller, and token count are in scope together.
Mistake 4: treating fallback as retry
Retrying an overloaded vendor on 429 amplifies the outage — you have added load to a service that just told you it is saturated. Real failover crosses vendor boundaries, which means the routing layer must know which models are interchangeable. That is a data problem, not a retry-policy problem.
Decision criteria: when to use which
Skip “it depends” and test your situation against three rules. If you expose public HTTP APIs to services you own, you need a traditional gateway for TLS, WAF, and per-consumer quotas. If you run two or more model providers, enforce a spend budget in real time, or hold prompt data under compliance review, you need an AI gateway. If you do both, you need both — the edge protects your services, the egress layer protects spend and uptime.
Then watch for these triggers, which mean you have outgrown a hand-rolled proxy:
- Provider-specific request or response shaping appears in more than one service.
- Changing a model or provider requires a deploy instead of a config edit.
- Multiple keys or teams, and no single query answers “who spent what.”
- Retry and backoff logic is duplicated across services and the copies disagree.
- A prompt-logging policy must satisfy an auditor, not just a developer.
The rule of thumb: the moment provider-specific code appears in a second service, extract the egress layer. Below that threshold a thin adapter is fine. Above it, you pay a maintenance tax on every model change. Our AI API gateway guide walks through the full feature set.
One key, many models: what a hosted AI gateway gives you
The architecture above is correct but not free to operate — someone must run the egress tier, keep the model registry current, and track every vendor release. For most teams under roughly ten engineers, a hosted relay is the better trade. qoraapi.com exposes many models from multiple vendors behind one OpenAI-compatible endpoint and one key, collapsing the egress column of the comparison table into a single integration.
- Vendor keys never touch your codebase. Your app holds one relay credential; provider credentials live behind the gateway, so rotation stops being a multi-service event.
- Model routing becomes a string. Move a workload between tiers by changing the
modelfield — no adapter code, no per-vendor SDK. - Cross-vendor fallback without building a health checker. When a provider degrades, the relay can serve from an equivalent model instead of returning a 429 to your user.
- One usage view instead of N dashboards. Token consumption across every model lands in one place — the prerequisite for per-feature cost work, covered in our AI API cost reduction guide.
- Streaming that behaves. OpenAI-compatible SSE pass-through, so existing client code works unchanged.
Note what a relay is not: it is not your edge. Keep the traditional gateway in front for user auth and WAF, keep the relay as your egress tier, and the boundary holds exactly as drawn — each layer owning what it was built for.
Frequently asked questions
Can I just use Kong or NGINX as my AI gateway?
For pure proxying, yes. For LLM-specific behavior you will end up writing plugins. You get no token metering and no capability-aware failover across vendors, and SSE breaks unless you disable buffering — which is on by default.
Do I need an AI gateway if I only use one provider?
Not for routing. You still benefit from token metering, keeping the vendor key out of your application, and controlling what prompt data gets logged. The value curve is non-linear: the second provider is where a gateway stops being nice-to-have.
Is an AI gateway a security boundary?
It is a credential boundary, not a user-auth boundary. It protects your provider keys and enforces spend, but it does not do WAF, bot mitigation, or tenant RBAC over your own resources. Keep user authentication at the API gateway and treat the AI gateway as an internal service.
Does adding an AI gateway hurt latency?
A well-built gateway adds single-digit milliseconds of routing overhead — noise next to a multi-second model call — and the semantic cache usually reduces median latency. The real risk is buffering in the streaming path, not the extra hop. Verify with time-to-first-token, since buffering is invisible in averages.
Conclusion
An API gateway and an AI gateway are not competitors. One governs who reaches your services; the other governs which model answers each prompt, what it costs, and what happens when a vendor fails. The overlap is shallow — auth and request limits, both of which the edge does better. The gaps are deep: token budgets, cross-vendor failover, streaming correctness, prompt-logging policy, and semantic caching cannot be bolted onto an HTTP proxy. For a public application that calls models, the default is both, in that order.
Leave a Reply