Self-Hosting an AI Gateway: Architecture, Scaling, and Ops

Self-hosting an AI gateway means running the proxy layer that sits between your applications and model providers on infrastructure you control. It buys you credential isolation, routing logic you can change in a single deploy, and per-request cost attribution that a black-box endpoint will never give you. It costs an on-call rotation, a provider schema-drift treadmill, and a stateful data plane that must not fall over. Below a few hundred million tokens a month, a managed gateway is usually the better trade; above that, or under a jurisdiction constraint you cannot negotiate away, self-hosting starts to win.

What a managed gateway gives you, and what it structurally cannot

A managed gateway – qoraapi.com being one example – collapses a lot of undifferentiated work into a single OpenAI-compatible endpoint: credentials handled by someone else, failover already wired, one invoice, one rate-limit surface, one place to look when a provider degrades. That is not marketing. If your routing policy is “use model X, and if it 429s use model Y,” rebuilding it buys you nothing.

The gaps appear when requirements stop being generic. A managed gateway cannot guarantee that a prompt body never leaves a specific region, because the payload is by definition in someone else’s process. It cannot route on your tenant taxonomy – enterprise on EU inference, free tier on the cheapest healthy provider, anything flagged medical pinned to a zero-retention endpoint. It cannot redact PII before egress or cache on your own key taxonomy, and it cannot let you survive a provider relationship ending without an application release.

Those are the four reasons teams self-host: data control, bespoke routing, cost transparency, and provider independence. Everything else is a proxy with extra steps.

The real cost of ownership

The infrastructure is the cheap part. The expensive part is that a gateway is a Tier-0 dependency for every LLM feature you ship, and it changes underneath you continuously: providers add parameters, deprecate parameters, change defaults, and quietly alter streaming chunk shapes. Adapter code churns constantly even when your own product is frozen.

Steady state for a moderately complex gateway is a quarter to a half of an engineer’s time, permanently – incident response, provider migrations, credential rotation, capacity work, config reviews. The first quarter costs more, and any provider migration costs more again. Teams that budget for the build and not the maintenance end up with a half-maintained gateway, which is worse than a managed one: it fails in ways nobody on the current team understands.

The component map

Every gateway decomposes into the same ten parts. Knowing which ones you are skipping is the difference between a working proxy and an outage.

• Credential vault. Provider keys must never live in application config. Store them with envelope encryption under a KMS key, scoped per environment, rotated on a schedule you have rehearsed. The important property is asymmetry: applications hold a gateway key, the gateway holds provider keys, so a leaked application key is revoked without touching provider credentials. See the AI API security checklist.
• Provider adapters. One module per provider, translating your canonical request into its native shape. This is the highest-churn code in the system, so test it hardest.
• Request normalisation. A canonical schema. OpenAI’s chat completions shape is the de facto lingua franca, with well-known leaks around tool calls and multimodal content.
• Routing policy. Rules over model alias, tenant, region, cost ceiling and provider health.
• Rate limiting. Per key, per tenant, per model. Token-aware, not just request-count-aware, or one long-context request bypasses your protection.
• Retry and failover. Error classification, retry budgets, jitter, and a circuit breaker per provider.
• Response cache. Exact-match at minimum; semantic if the workload justifies it.
• Usage metering. Token counts, cost and latency attributed to a tenant and a feature.
• Observability. Per-request traces with prompt and response redaction, plus aggregate metrics.
• Admin API. Keys, budgets, config versions, model catalogue. The control plane.

The split that matters most is control plane versus data plane. The control plane can be down for an hour and nothing breaks, provided the data plane serves from its last-known-good configuration. Build that property in on day one; retrofitting it during an incident is not possible.

Deployment topologies compared

Topology	Added latency	Blast radius	Upgrade path	Where it fits
Central cluster	One in-region network hop plus TLS and proxy work	Everything that talks to the gateway	One deploy changes behaviour fleet-wide, instantly	The default. Fix a provider outage once for all consumers.
Sidecar per pod	Loopback only	One pod, but a bad config needs every pod rolled to fix	Roll every workload; config distribution becomes the hard problem	Strong per-tenant network isolation requirements in a Kubernetes-heavy shop
In-process library or SDK	None	One process	Every application redeploys, in every language	Single-language monoculture, one owning team, no cross-team consumers

Default to a central cluster. It is the only topology where a routing fix, a credential rotation or a provider failover lands everywhere at once. Sidecars sound appealing until you realise per-pod connection pools multiply your provider connection count by your replica count – exactly the thing providers rate-limit you on. In-process libraries work for a single-language shop, but you reimplement config, metering and observability once per language, and you lose failover without an application release – the main reason you built a gateway.

Stateless versus stateful: keeping counters out of the request path

The request path should be stateless wherever it can be. Four things genuinely cannot be, and each needs a deliberate failure policy rather than a default.

Stateful component	Consistency needed	Sensible failure policy
Rate-limit counters	Strong per key, approximate globally	Fail open, with alerting. Rejecting all traffic because Redis blipped is worse than a short burst of overspend.
Budget and spend counters	Strong if the cap is contractual, eventual if it is advisory	Fail closed only when the number is a hard quota. Batch increments for advisory counters and accept bounded overshoot.
Response cache	None	A miss is the normal path. A cache outage must degrade to full origin traffic, never to 5xx.
Circuit-breaker health	Per instance is enough	Local state, converges in seconds. Never make it a shared dependency.

The discipline is the same in all four cases: bound the round trip. A rate-limit check that can block for two seconds is worse than no rate limiter, because it converts a load problem into a fleet-wide latency problem. Give the counter a hard timeout in the tens of milliseconds and decide in advance what happens when it expires.

The check must be one atomic round trip: read-modify-write from application code races under concurrency and lets bursts through.

-- token bucket, atomic in one round trip, no read-modify-write race
-- KEYS[1] = bucket key   ARGV = capacity, refill_per_sec, now_ms, cost
local capacity = tonumber(ARGV[1])
local refill   = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local cost     = tonumber(ARGV[4])

local b      = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(b[1]) or capacity
local ts     = tonumber(b[2]) or now

tokens = math.min(capacity, tokens + (now - ts) / 1000 * refill)

if tokens < cost then
  redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
  redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / refill * 1000))
  -- second return value is the retry-after hint in milliseconds
  return { 0, math.ceil((cost - tokens) / refill * 1000) }
end

redis.call("HMSET", KEYS[1], "tokens", tokens - cost, "ts", now)
redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / refill * 1000))
return { 1, 0 }

A routing policy you can actually run

Policy belongs in declarative config, versioned and validated in CI, not in code branches. A workable shape:

# gateway/policies/chat-default.yaml
version: 3
route: chat-default

match:
  model_alias: [ "chat-fast", "chat-balanced" ]

providers:
  - id: openai-primary
    adapter: openai
    model: gpt-4.1-mini
    weight: 100
    timeout:
      connect_ms: 400
      first_byte_ms: 2500
      total_ms: 30000

  - id: anthropic-fallback
    adapter: anthropic
    model: claude-sonnet-4
    weight: 0                 # failover target only, never load-balanced
    timeout:
      connect_ms: 400
      first_byte_ms: 3000
      total_ms: 45000

  - id: bedrock-eu
    adapter: bedrock
    region: eu-central-1
    model: meta.llama3-70b
    weight: 0
    timeout:
      connect_ms: 600
      first_byte_ms: 4000
      total_ms: 60000

retry:
  max_attempts: 3
  budget_ratio: 0.10          # retries may never exceed 10% of total traffic
  backoff: exponential_jitter
  retry_on: [ "connect_timeout", "first_byte_timeout", "http_429", "http_5xx" ]
  never_retry_on: [ "http_400", "http_401", "http_403", "content_filter" ]

circuit_breaker:
  error_rate_threshold: 0.50
  min_requests: 20
  open_seconds: 15

cache:
  exact_match: true
  ttl_seconds: 3600
  key: [ model_alias, messages, temperature, tenant_id ]

limits:
  tenant_rpm: 600
  tenant_tpm: 400000

Two details carry most of the value. First, budget_ratio: without it, retries are unbounded amplification, as documented in the multi-provider failover playbook. Second, weight: 0 on the fallbacks – a fallback receiving steady traffic is not a fallback but a second primary with worse cost characteristics and no warm-up guarantee.

Scaling the data plane

Connection pooling is the whole game. Keepalive pools must be sized to peak concurrency, not to instance count, and each provider needs its own pool because their latency and rate-limit profiles differ. HTTP/2 multiplexing helps, but a single connection carries a bounded number of concurrent streams – typically around a hundred – and beyond that requests queue invisibly. An undersized pool shows up as p99 latency that looks like provider slowness but is local queueing.

Streaming changes the capacity model. An SSE response holds a connection for the entire generation. An instance serving five hundred concurrent streams is nearly idle on CPU and bounded by memory buffers and file descriptors, not compute. Request-per-second is therefore the wrong autoscaling signal; scale on in-flight streams. Disable response buffering at every hop – proxy_buffering off in nginx, X-Accel-Buffering: no from the proxy – or your streaming endpoint will deliver the whole answer in one burst after a ten-second stall. Never compress an event stream.

import asyncio, time, httpx
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

UPSTREAM = {
    "openai":    "https://api.openai.com/v1/chat/completions",
    "anthropic": "https://api.anthropic.com/v1/messages",
}

# One pool per provider. pool=0.1 is the admission-control valve: if no
# connection is free within 100ms we shed load instead of queueing forever.
POOLS = {
    "openai": httpx.AsyncClient(
        limits=httpx.Limits(max_connections=256, max_keepalive_connections=128),
        timeout=httpx.Timeout(connect=0.4, read=30.0, write=5.0, pool=0.1),
    ),
    "anthropic": httpx.AsyncClient(
        limits=httpx.Limits(max_connections=128, max_keepalive_connections=64),
        timeout=httpx.Timeout(connect=0.4, read=45.0, write=5.0, pool=0.1),
    ),
}

app = FastAPI()

@app.post("/v1/chat/completions")
async def chat(request: Request):
    body     = await request.json()
    tenant   = request.headers["x-tenant-id"]
    provider = route(body)          # policy lookup, no I/O

    if not body.get("stream"):
        return await collect(provider, body, tenant)

    async def relay():
        started = time.monotonic()
        client  = POOLS[provider]
        async with client.stream(
            "POST", UPSTREAM[provider], json=body,
            headers={"Authorization": "Bearer " + await keyring.get(provider)},
        ) as up:
            up.raise_for_status()
            async for chunk in up.aiter_raw():
                if await request.is_disconnected():
                    break            # stop paying for tokens nobody will read
                yield chunk
        # fire-and-forget: metering must never sit on the response path
        asyncio.create_task(emit_usage(tenant, provider, time.monotonic() - started))

    return StreamingResponse(
        relay(),
        media_type="text/event-stream",
        headers={"cache-control": "no-store", "x-accel-buffering": "no"},
    )

The is_disconnected check is not an optimisation. Without it, a user closing a tab leaves the upstream generation running, and you pay for every token of it.

The sticky-session trap. The temptation with long-lived streams is to pin a client to an instance by cookie or source IP. Do not. Stickiness destroys load balancing, strands in-flight streams on instances you are draining, and turns every deploy into a rolling brownout. The alternative is a stateless router plus graceful shutdown: on SIGTERM stop accepting new connections, let in-flight streams finish up to a deadline, then exit. Set the load balancer idle timeout above your maximum generation time, or long streams get severed mid-sentence by a healthy-looking proxy.

Backpressure has three layers, and you need all three: connection limits at the load balancer, a concurrency semaphore per instance, and a per-tenant concurrency cap. When saturated, reject with 429 and a Retry-After rather than queueing: queueing converts a capacity problem into a timeout storm, and timeouts consume capacity while producing nothing. The pool=0.1 timeout above is the mechanism – deliberate load shedding, not a misconfiguration.

Failure modes, in the order they will bite you

1. Config rollout that breaks all traffic. The most common self-inflicted gateway outage by a wide margin: one inverted condition, one typo in a provider ID, or a schema change that silently drops a field, and 100% of traffic fails at once. Validate config against a strict schema in CI and reject unknown fields; roll out 1% then 10% then 100% with automatic rollback on error-rate delta; and keep last-known-good config in the data plane so a control-plane failure cannot take down serving.
2. Retry storms. A provider starts 429ing, every instance retries, the retries multiply load threefold, the provider pushes back harder, and a transient degradation becomes a sustained outage. Fixes: a retry budget as a fraction of total traffic, full-jitter backoff, a per-provider circuit breaker, and a hard rule never to retry on 400, 401, 403 or content-filter responses – those are deterministic and will fail identically three times.
3. Cache stampede. A popular prompt’s entry expires, two hundred concurrent requests miss simultaneously, and all two hundred hit the provider with an identical payload. Use single-flight: the first request fills the entry, the rest await the same future. Add TTL jitter so entries do not expire in lockstep, and a stale-while-revalidate window so a refresh never blocks a reader.
4. Hot partitions in the rate limiter. One tenant with a shared bucket saturates a Redis shard and adds latency to every other tenant on it, because a single global counter is a single hot shard. Shard by tenant, give any tenant large enough to saturate one shard its own bucket with a longer refill window, and monitor per-shard latency rather than the aggregate.
5. Provider schema drift. A provider adds a response field, changes a default, or deprecates a parameter, and your adapter silently drops it – behaviour changes without an error. Use contract tests against recorded fixtures per provider version, count unrecognised response fields and alert when the counter moves, and run strict mode in staging that errors on unknown fields.

Notice what is absent from this list: provider downtime. Providers are more reliable than the code you write on top of them, and the outages you remember will be your own.

A worked TCO example

An illustrative calculation with visible arithmetic, not a benchmark or a quote. Substitute your own numbers; the shape of the result is what matters.

• Volume. 5,000,000 requests/day. Mean request payload 4 KB, mean response payload 8 KB, so 12 KB per request.
• Peak concurrency. Mean rate is 5,000,000 / 86,400 = 58 requests/second. With a 3x diurnal peak that is 174 requests/second. If 30% of traffic streams for a mean 8 seconds, peak concurrent streams are 174 x 0.30 x 8 = 418.
• Compute. At 250 concurrent streams per instance for headroom, 418 / 250 = 1.7, so 2 instances for peak plus 1 for AZ and deploy redundancy = 3 instances. A 4 vCPU / 8 GB instance at $0.19/hour is 0.19 x 730 = $139/month. Three of them: $417/month.
• Egress. 5,000,000 x 12 KB = 60 GB/day, or 1,800 GB/month. At $0.09/GB: $162/month.
• Load balancer and NAT data processing: $65/month.
• State store. A managed Redis with a replica: $80/month.
• Observability. Full trace capture would be 5,000,000 x 1.5 KB = 7.5 GB/day. Sample 10% instead: 750 MB/day, about 22 GB/month, roughly $7 at $0.30/GB ingest, plus $60 for metrics and logs. Call it $70/month.
• Infrastructure subtotal: 417 + 162 + 65 + 80 + 70 = $794/month.
• Engineering. 0.35 FTE at $200,000 fully loaded = $70,000/year = $5,833/month.
• Total: $794 + $5,833 = $6,627/month, or $0.044 per 1,000 requests at 150,000,000 requests/month.

The ratio is the point: infrastructure is roughly one seventh of the cost and people are six sevenths. Two consequences follow. First, “it is cheaper” is not a valid argument for self-hosting – the dominant term barely moves between 5M and 50M requests/day, so per-request economics improve sharply with volume while absolute cost hardly changes. Second, compare the fully loaded $6,627 against what a managed gateway costs at this volume. If the managed fee lands in the same range, self-hosting is a bad trade on cash terms alone.

When self-hosting is worth it, and when it is not

Self-host when a regulator or customer contract requires payloads to stay inside a specific VPC or jurisdiction and no vendor will commit to that contractually. When routing must be genuinely bespoke – cost-aware tiering, per-tenant provider pinning, pre-egress redaction, region-pinned inference for a subset of traffic. When you need to switch providers without an application release, which is an architectural property, not a cost one.

Do not self-host when you are pre-product-market-fit and routing is “one provider with occasional failover.” When nobody will be paged for a proxy at 3am. When you cannot commit a quarter of an engineer’s time permanently, because a half-maintained gateway is worse than a managed one – it fails in ways nobody currently employed understands. And when the only motivation is cost, which the arithmetic above shows rarely survives contact with the staffing number.

The strongest position is not to pick one. Keep your application speaking a single canonical, OpenAI-compatible schema, put credentials and routing behind an interface you own, and treat the gateway as a swappable component. Start managed, and migrate when a concrete requirement – not a vibe – forces you. Teams that self-hosted for a compliance obligation are happy with the decision; teams that self-hosted to save money usually are not. For a wider survey of the managed option, see the AI API gateway guide.

Frequently asked questions

How much latency does a self-hosted gateway actually add?

One in-region network hop plus TLS termination and proxy work. Provider time-to-first-token dominates the end-to-end number, so if proxy overhead is more than a small fraction of total TTFB, the cause is almost always a state lookup on the request path: an unbounded Redis call, a synchronous config fetch, or uncached DNS resolution. Instrument the proxy span separately from the upstream span; if you cannot separate them, you have an observability problem before a latency problem.

Do I need three instances, or is two enough?

Two instances survive a single failure but run at 50% capacity each at peak, so you pay for redundancy you can never use. Three runs each at 67% and absorbs both an AZ loss and a rolling deploy. If your gateway fronts a chat feature, two is defensible; anything with a revenue SLA needs three.

Should the response cache live in the gateway or the application?

Exact-match caching belongs in the gateway: the key is derivable from the request and the benefit is shared across consumers. Semantic caching is a product decision – the embedding model, similarity threshold and staleness window depend on your workload – so it belongs where those can be tuned per feature. The mechanics are covered in the semantic caching writeup. Either way, a cache miss must be an ordinary code path, never an error.

How do I handle provider-specific parameters the canonical schema does not have?

An explicit, allowlisted passthrough bag – something like provider_options – scoped to a single request. Never a blanket passthrough of arbitrary fields: it lets application code set parameters that violate your routing invariants, and it is how you end up with a request pinning a model the router thought it was choosing. The allowlist is small and grows by review, which is the point.

What should I instrument first?

Four things, in order: a per-request trace carrying provider, model, attempt number and outcome; upstream time-to-first-byte and total duration as separate histograms; prompt and completion token counts attributed to tenant; and a retry counter broken down by reason. Those answer almost every question you will have during an incident. Prompt and response bodies are useful in staging and dangerous in production – see the observability guide for keeping the useful parts while keeping payloads out of your log store.

Conclusion

Self-hosting an AI gateway is a straightforward engineering problem wrapped in a hard organisational one. The engineering is well understood: normalise requests, isolate credentials, keep the request path stateless, bound every state lookup, control retries with a budget, and make the control plane incapable of taking down the data plane. The hard part is that you are signing up for permanent ownership of a Tier-0 service whose dependencies change every few weeks.

Decide on requirements, not on cost. If a jurisdiction constraint, a bespoke routing rule or a genuine provider-independence requirement forces your hand, self-host and staff it properly – a quarter of an engineer, permanently, not a sprint. If none apply, run managed, keep your client interface canonical, and preserve your ability to move later. That optionality is worth more than the infrastructure savings, and it is the one decision you cannot retrofit cheaply once application code has learned a vendor’s quirks.

Related reading

• What Is an AI API Gateway? A Practical Guide for Developers
• Ultimate Guide: How to Choose the Best AI API Gateway in 2026
• AI Gateway vs API Gateway: Key Differences and When to Use Each
• How to Build a Multi-Provider AI Failover Layer for 99.9% Uptime