Category: AI API

Serverless fits a subset of LLM workloads and fails the rest, and the deciding variable is almost always wall-clock time. Classification, embedding, and short single-shot completions that finish in under ten seconds are cheaper and simpler on a FaaS platform than on a container. Long generations, agent loops, and extended-reasoning calls are not: the platform kills them mid-response, and you pay for tokens the client never received. Decide per endpoint, not per architecture.

The mismatch is structural

The serverless contract assumes work that is short, bursty, stateless, and killable at any instant. LLM calls are long-tailed, incremental, and stateful across turns. Each mismatch maps to a constraint you cannot negotiate away:

• Execution time limits. A hard platform ceiling, with no graceful shutdown.
• Response size limits. API Gateway caps payloads at 10 MB; Lambda response streaming at 20 MB.
• No durable local state. /tmp and in-process memory are recycled with the environment.
• Cold starts. A fresh environment pays runtime init, dependency loading, and a TLS handshake first.
• No guaranteed connection reuse. Pooling happens only if the platform reuses the same warm environment.
• At-least-once invocation. Queues and client retries can run your handler twice.

What statelessness costs a chat product

Multi-turn chat needs history, which makes the function stateful by proxy. Pushing it to Redis or Postgres adds a round trip to every turn and turns a pure compute tier into a compute-plus-datastore tier. If that store lives in a VPC, the function’s network path and cold start are coupled to it too. Worse, prompt-prefix caching depends on an exact prefix, so a reordered history invalidates the cache you were counting on.

Platform constraints, compared

Three families matter, and the differences that decide the design are duration, streaming, and connection reuse across requests.

Property	Classic FaaS (AWS Lambda-style)	Edge runtime (Cloudflare Workers-style)	Container platform (Fargate, Cloud Run, Fly.io)
Max duration	15 minutes per invocation; 29 seconds if fronted by API Gateway	No fixed wall-clock limit while a response stream is open; CPU time is capped separately, commonly 30 seconds default and extendable	Cloud Run defaults to 300 seconds per request, configurable to 60 minutes; long-running services have no per-request cap
Streaming support	Only via Function URLs with response streaming; never through API Gateway	Native and first-class; the runtime is built around ReadableStream	Native; the only family that streams reliably through a normal load balancer
Outbound connection reuse	Within a warm execution environment only, and only if the client lives outside the handler	Within an isolate, best-effort; isolates are recycled aggressively	Real keep-alive pools that survive across requests, with tunable pool sizes
Cold start profile	Hundreds of milliseconds to several seconds, driven by runtime, bundle size, and VPC configuration	Low single-digit milliseconds for isolate creation; the real cost is the first outbound TLS handshake in a new isolate	Seconds from zero, but eliminated by keeping a minimum instance count
Runtime limits	Node, Python, Java, .NET, Go, Rust; 128 MB to 10 GB memory; 50 MB zipped deployment package; no inbound sockets	V8 isolates with Web Platform APIs; 128 MB memory; no filesystem; subrequest counts capped per invocation	Whatever fits in a container; no meaningful runtime restrictions
Best fit	Bursty, short, spiky calls: classification, embedding, single-shot completions	Latency-sensitive short calls and streaming proxies where compute is mostly waiting	Sustained concurrency, long generations, agent loops, in-process prompt caches

The column that quietly decides most architectures is outbound connection reuse. A container keeps a TLS session to the provider open for hours; a function keeps one only while its environment survives, and an edge isolate only until eviction. Duration, memory, and runtime can be worked around. That one cannot, which is why serverless LLM deployments so often route through a long-lived intermediary.

Execution time limits are the design constraint

Treat the platform ceiling as the primary filter. A workable rule of thumb: compare your p95 end-to-end provider latency against roughly a quarter of the platform’s wall-clock budget, because you need headroom for cold start, retries, and the tail. If the budget is 30 seconds, do not inline anything with a p95 above seven or eight seconds. At 15 minutes, the threshold moves to three or four minutes.

Workloads that fit

• Short completions. Extraction, rewriting, labelling, short structured output.
• Classification and routing. Low single-digit latency, deterministic token counts.
• Embeddings. No streaming, bounded output, per-item billing. The cleanest fit here.
• Small RAG answers. Retrieve, stuff a bounded context, generate briefly — if retrieval is not a slow VPC hop.

Workloads that do not fit

• Long generations. 4,000 tokens at 40 per second is 100 seconds of streaming: fine on an edge runtime, fatal behind a 29-second integration timeout.
• Deep reasoning. Time-to-first-token alone can exceed a FaaS front door’s entire budget.
• Batch jobs. Per-item overhead dominates; these belong on workers that amortise startup.
• Agent loops. A ten-step loop of tool calls multiplies the odds of crossing the ceiling.

A well-behaved serverless LLM handler builds its HTTP client at module scope so the pool survives across warm invocations, sets timeouts per phase, and records when it exceeded its own budget — that log line is the leading signal that the workload is outgrowing the platform.

import os
import time
import httpx

# Module scope. A client created inside the handler pays a fresh TCP and
# TLS handshake on every single invocation.
_client = httpx.Client(
    base_url="https://api.qoraapi.com",
    timeout=httpx.Timeout(connect=2.0, read=8.0, write=2.0, pool=1.0),
    limits=httpx.Limits(max_keepalive_connections=20, max_connections=50),
    headers={"authorization": "Bearer " + os.environ["QORA_API_KEY"]},
)

BUDGET_S = 8.0  # the p99 we are willing to absorb inside the function

def classify(text: str) -> str:
    started = time.monotonic()
    resp = _client.post(
        "/v1/chat/completions",
        json={
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "system", "content": "Reply with exactly one label."},
                {"role": "user", "content": text},
            ],
            "max_tokens": 8,
            "temperature": 0,
        },
    )
    resp.raise_for_status()
    elapsed = time.monotonic() - started
    if elapsed > BUDGET_S:
        # Succeeded, but only because the platform had headroom left.
        # Emit this as a metric: it is the leading indicator of a bad fit.
        print("over-budget call: %.2fs" % elapsed)
    return resp.json()["choices"][0]["message"]["content"].strip()

Streaming from serverless

Streaming exists to move time-to-first-token from seconds to hundreds of milliseconds. Buffering the whole response inside the function destroys that property and adds a failure mode: the entire output must fit in memory and inside the payload limit. If you are going to buffer, you have chosen a synchronous API and should use the job-and-webhook pattern instead.

Passing a stream through a function

The mechanics differ by platform but the shape does not: read the upstream body as a stream, write each chunk to the response stream as it arrives, never accumulate. On Lambda that means response streaming through a Function URL using the streaming handler signature — API Gateway cannot do it, and its payload limit and 29-second integration timeout make the question moot. On edge runtimes it is the default: return a ReadableStream and the runtime forwards chunks as produced.

The heartbeat problem

Every hop between your function and the client has an idle timer, and all of them are shorter than a slow generation. An ALB idles a connection out at 60 seconds by default. nginx’s proxy_read_timeout also defaults to 60. The timer measures inactivity, not total duration, so the fix is to ensure no gap ever approaches it: emit an SSE comment line every 15 seconds. Comment lines begin with a colon and are ignored by every conforming parser, so the cost is a few bytes and the benefit is surviving a model that pauses before its first token.

Two details bite in production. Buffering proxies must be disabled — send X-Accel-Buffering: no and Cache-Control: no-cache, no-transform, or an intermediary holds your chunks until it has enough to flush, reintroducing the latency you removed. And compression must be off; a gzip filter with a 1 KB buffer will sit on your first token until the second arrives.

Cold starts: what actually causes them

Cold start is not one delay. It is four, and they are separable.

• Runtime initialisation. Node and Python are fast; a JVM is not.
• Dependency loading. A fat SDK that drags in half of AWS adds hundreds of milliseconds.
• TLS handshake to the provider. A client built per invocation pays a full handshake before the request is sent.
• DNS resolution. Often ignored, occasionally dominant — especially in a VPC, where queries hit the VPC resolver.

Mitigations that work

• Smaller bundles. Ship the HTTP client you need, not a framework. Costs nothing at runtime.
• Module-scope clients. Build them outside the handler so they survive between invocations.
• Provisioned concurrency. Environments already past runtime init and holding an open provider connection. The only fix that removes the handshake cost rather than hiding it.
• More memory, on Lambda. CPU scales with memory, so this genuinely shortens CPU-bound init.

What is cargo cult

A scheduled ping every five minutes to keep the function warm does not reliably work. It warms one execution environment, it does not control which environment serves the next real request, and under concurrency you need as many warm environments as simultaneous requests. Warm-up plugins on a schedule share the flaw. The old claim that VPC configuration adds ten seconds to every cold start is obsolete — hyperplane network interfaces fixed it — though VPC still adds latency through NAT hops and DNS.

resource "aws_lambda_function" "llm_proxy" {
  function_name = "llm-proxy"
  runtime       = "nodejs20.x"
  handler       = "index.handler"

  # CPU scales with memory, so this also shortens CPU-bound cold start.
  memory_size = 1024
  # A hard ceiling. Exceeding it terminates the invocation mid-write.
  timeout = 30

  filename         = data.archive_file.bundle.output_path
  source_code_hash = data.archive_file.bundle.output_base64sha256

  environment {
    variables = {
      # An ARN, not a key. The function fetches and caches the secret itself.
      QORA_API_KEY_SECRET_ARN = aws_secretsmanager_secret.qora.arn
      UPSTREAM_TIMEOUT_MS     = "20000"
    }
  }

  # Pre-initialised environments: past runtime init and connection-warm.
  # Billed whether or not they serve traffic.
  provisioned_concurrency_config {
    provisioned_concurrent_executions = 5
  }
}

resource "aws_lambda_alias" "live" {
  name             = "live"
  function_name    = aws_lambda_function.llm_proxy.function_name
  function_version = "$LATEST"
}

# 29000 is the maximum. API Gateway will not let you raise it.
resource "aws_api_gateway_integration" "proxy" {
  http_method             = "POST"
  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  timeout_milliseconds    = 29000
  uri                     = aws_lambda_function.llm_proxy.invoke_arn
}

Connection reuse and TLS

A TLS handshake to a provider endpoint costs roughly two network round trips: TCP plus TLS negotiation, with TLS 1.3 getting a resumed session down to one. Call it 40 milliseconds to a region a few hundred kilometres away. Against a 20-second generation that is noise. Against a 300-millisecond classification call it is 13 percent of your latency, and if the function also pays runtime init and dependency loading, it can be the second-largest component of the request.

The arithmetic inverts the usual intuition: the shorter your LLM call, the more a cold handshake matters proportionally — which is exactly the workload serverless is supposed to be good at. The fix takes two forms. Where the platform allows it, keep the client at module scope so warm environments reuse the socket; that is free and you should always do it. Where it does not, route through a gateway holding a pooled upstream connection, so your function talks to one nearby endpoint instead of handshaking with a distant provider region. That also removes per-provider pools from your code, and is a large part of why teams put a gateway in front of their providers rather than calling them directly.

Secrets: environment variables are a liability

A provider key in a function’s environment variables is readable by anyone holding lambda:GetFunctionConfiguration, stored in plaintext in your Terraform state, leaked into crash dumps and error payloads, and impossible to rotate without redeploying every function that references it. With forty functions across four environments that is a coordinated fleet-wide deploy, which in practice means the key never rotates.

• Secret manager with a cached fetch. Store the ARN, fetch on first use, cache in module scope, refresh on a TTL.
• Short-lived credentials. Use the platform identity — IAM role, workload identity, OIDC — to mint a short-lived token.
• One scoped key to a gateway. The function holds a single revocable key; the gateway holds the provider credentials. Rotation is one change in one place.

This is the same reasoning behind any sound LLM API security posture: minimise long-lived secrets, and make the ones that remain cheap to rotate.

The cost model: where the crossover is

Serverless wins on spiky traffic and loses on sustained traffic, and the crossover is calculable. Take published on-demand rates: Lambda at $0.0000166667 per GB-second plus $0.20 per million requests, Fargate at $0.04048 per vCPU-hour plus $0.004445 per GB-hour.

Worked example. Each request runs 4 seconds at 512 MB, so it consumes 4 x 0.5 = 2 GB-seconds. Per-request cost is 2 x $0.0000166667 = $0.00003333, plus $0.0000002 for the request itself, totalling $0.00003353.

• Spiky, 50,000 requests per day. Lambda: 50,000 x $0.00003353 = $1.68/day, about $50/month. Peak load is near 14 concurrent calls, so containers need two 1 vCPU / 2 GB tasks: 2 x ($0.04048 + 2 x $0.004445) x 730 = $72, plus ~$20 for an ALB. Roughly $92. Serverless wins.
• Steady, 1,000,000 requests per day. Lambda: 1,000,000 x $0.00003353 = $33.53/day, about $1,006/month. That averages 11.6 requests per second, roughly 46 concurrent calls; at a conservative 15 per 1 vCPU task, four tasks suffice: 4 x $36.04 = $144, plus $25 for an ALB. Roughly $170. Containers win by about six times.

The crossover sits where Lambda’s monthly bill equals the container floor of about $170: roughly 5.1 million requests per month, or 170,000 per day — a sustained two requests per second. Below that, serverless is cheaper. Above it, you pay a per-invocation premium for elasticity you no longer need.

Profile	Requests per day	Serverless monthly	Container monthly	Cheaper option
Spiky, 4-hour peak window	50,000	~$50	~$92	Serverless
Sustained, flat	1,000,000	~$1,006	~$170	Container, by ~6x
Crossover	~170,000	~$170	~$170	Indifferent on cost; decide on latency and control

Two caveats shift the crossover toward containers. Provisioned concurrency is billed while idle, adding to the serverless column without adding throughput. And if your handler is CPU-bound rather than waiting on the network, the concurrent calls a container task can serve drops sharply.

Failure modes specific to serverless

Silent truncation mid-stream

Once the first SSE chunk is flushed, the HTTP status is committed. If the function then hits its timeout you cannot send a 500 — the client already has a 200 and a partial body. It sees a stream that simply stops, indistinguishable from a network blip, and your application may store a half-finished answer as complete. The fix is an in-band terminal marker: emit an explicit event: truncated frame when the read loop exits without a [DONE] or finish_reason, and make clients treat a stream that closes without one as a failure regardless of status code.

Duplicate execution on retry

Async invocations retry on failure, queues redeliver after a visibility timeout, and API clients retry on connection errors. A retried LLM call is a second billable generation and, if it writes to your database, a duplicated record. You cannot make the provider call idempotent, so idempotency has to live at your boundary: an idempotency key, a short-lived record of completed work, and a client contract that treats a repeat as a lookup. This is the discipline that makes asynchronous job delivery safe, and it applies equally to a synchronous endpoint that retries.

Losing the response when the client disconnects

When the browser tab closes, the function does not necessarily stop. On some runtimes the invocation runs to completion and you pay for every token. On others the runtime aborts the handler, and whether that abort reaches the in-flight upstream request depends entirely on your code. Propagate it explicitly: listen on the request’s abort signal and cancel the upstream fetch, so a disconnected client stops the generation instead of billing you for output nobody reads.

VPC and networking surprises

Placing a function in a VPC to reach a database adds a NAT hop to every outbound provider call: latency plus a per-gigabyte data processing charge. It also changes DNS resolution, so the first lookup in a cold environment can be slower than expected. If the function only needs the public internet and a database with a public endpoint, do not put it in a VPC. If it must, budget the NAT cost into the arithmetic above.

A streaming LLM endpoint on an edge runtime

This handler covers the two cases that break naive implementations: the deadline, and the client walking away. It never buffers, heartbeats on an interval, distinguishes an error before the first byte (where a real status code is still available) from one after (where only an in-band frame is), and aborts upstream on disconnect.

interface Env {
  QORA_API_KEY: string;
}

const UPSTREAM = "https://api.qoraapi.com/v1/chat/completions";
const DEADLINE_MS = 90_000;   // our own budget, below the platform ceiling
const HEARTBEAT_MS = 15_000;  // comfortably under a 60s proxy idle timeout

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== "POST") {
      return new Response("method not allowed", { status: 405 });
    }

    const payload = await request.text();
    const upstreamAbort = new AbortController();

    // Client disconnect must cancel the upstream generation, not just the response.
    request.signal.addEventListener("abort", () => {
      upstreamAbort.abort("client-disconnect");
    });

    const deadline = setTimeout(() => {
      upstreamAbort.abort("deadline-exceeded");
    }, DEADLINE_MS);

    let upstream: Response;
    try {
      upstream = await fetch(UPSTREAM, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          authorization: "Bearer " + env.QORA_API_KEY,
        },
        body: payload,
        signal: upstreamAbort.signal,
      });
    } catch (err) {
      clearTimeout(deadline);
      // Nothing has been flushed yet, so a real status code is still available.
      return new Response(JSON.stringify({ error: String(err) }), {
        status: 504,
        headers: { "content-type": "application/json" },
      });
    }

    if (!upstream.ok || upstream.body === null) {
      clearTimeout(deadline);
      return new Response(upstream.body, { status: upstream.status });
    }

    const encoder = new TextEncoder();
    const decoder = new TextDecoder();

    const stream = new ReadableStream<Uint8Array>({
      async start(controller) {
        const reader = upstream.body!.getReader();
        const heartbeat = setInterval(() => {
          // SSE comment line: ignored by parsers, resets every idle timer.
          controller.enqueue(encoder.encode(": keep-alive\n\n"));
        }, HEARTBEAT_MS);

        let tail = "";
        let sawTerminal = false;

        try {
          for (;;) {
            const { done, value } = await reader.read();
            if (done) break;
            tail = (tail + decoder.decode(value, { stream: true })).slice(-256);
            controller.enqueue(value);
            if (tail.includes("[DONE]") || tail.includes('"finish_reason"')) {
              sawTerminal = true;
            }
          }
        } catch (err) {
          // Status is already committed. The only channel left is in-band.
          const frame = JSON.stringify({ message: String(err) });
          controller.enqueue(encoder.encode("event: error\ndata: " + frame + "\n\n"));
        } finally {
          clearInterval(heartbeat);
          clearTimeout(deadline);
          if (!sawTerminal) {
            controller.enqueue(
              encoder.encode(
                'event: truncated\ndata: {"reason":"upstream ended without a terminal frame"}\n\n'
              )
            );
          }
          controller.close();
        }
      },

      cancel(reason) {
        // Response stream cancelled downstream: stop paying for the generation.
        upstreamAbort.abort(String(reason));
      },
    });

    return new Response(stream, {
      headers: {
        "content-type": "text/event-stream; charset=utf-8",
        "cache-control": "no-cache, no-transform",
        "x-accel-buffering": "no",
      },
    });
  },
};

Two things this handler deliberately does not do. It does not retry the upstream call, because a retry after partial output would append a second generation to the first; retries belong before the first byte or in a separate job. And it does not resolve a model to a provider — that is the gateway’s job, and keeping it there means failover across providers happens without redeploying the function.

When not to use serverless, and where I would default

Five conditions disqualify it, and any one of them is sufficient.

• Long generations. If p95 time-to-last-token exceeds a quarter of the ceiling, you are one bad tail latency from truncation.
• Sustained high concurrency. Past two requests per second you pay an elasticity premium for capacity you could buy outright.
• Heavy in-process state. A prompt cache, embedding index, or reranker — anything whose value depends on state surviving between requests.
• GPU needs. Cold-starting a GPU costs more than the request you were serving.
• A durable provider connection. A long-lived multiplexed connection or websocket upstream rules serverless out.

So: default to serverless for embedding, classification, extraction, short single-shot completions, and the generation half of a small RAG pipeline — anything with a p95 under about eight seconds and bounded output. You get near-zero fixed cost, trivial scaling, and no capacity planning, and the cold-start TLS penalty is manageable with a module-scope client. Default to containers for anything conversational at scale, anything streaming more than a few hundred tokens, anything with an agent loop, and anything where a local cache changes your cost profile. The moment you configure provisioned concurrency to hit a latency SLO, you are paying the container floor while still accepting serverless constraints. Time to move.

The hybrid that actually ships is a thin serverless or edge tier in front of a long-lived worker tier, with a gateway between them holding upstream credentials and pooled connections. The front tier absorbs bursts and terminates client connections; the worker tier owns the slow, stateful, expensive work.

Frequently asked questions

Can I just raise the function timeout?

Up to the platform maximum, yes, and it is often the right first move — Lambda allows 15 minutes per invocation. What you cannot raise is the timeout of the layer in front of it. API Gateway’s 29-second integration timeout is hard, an ALB’s 60-second idle timeout is configurable but bounded, and Cloudflare returns a 524 if an origin does not respond in time. Raising the function timeout only helps if the path bypasses those layers, for example a Function URL with response streaming. Measure the smallest ceiling in the path, not the largest.

Does provisioned concurrency solve cold starts?

It solves the parts you can pre-warm: runtime init, dependency loading, and the TLS handshake if your client lives at module scope. It does not solve anything downstream — provider-side queueing, model warm-up, or your database’s connection establishment. It is also billed whether or not it serves traffic, which makes it the clearest signal that a workload has outgrown serverless.

Can I stream from AWS Lambda?

Yes, but only through a Function URL with response streaming, using the streaming handler signature rather than the standard one. API Gateway cannot stream responses, and its 10 MB payload limit applies to the buffered case. Even with response streaming, plan for the 20 MB soft response limit and for the fact that once you write the first chunk your status code is fixed, so mid-stream failures must be signalled in-band. If streaming is central to your product, an edge runtime fits better than Lambda.

Should I put my provider API key in the function environment?

For a prototype, fine. Past a couple of functions, no. Environment variables are readable by anyone with configuration-read permission, stored in plaintext in your infrastructure state, visible in error payloads, and require a fleet-wide redeploy to rotate. Store the secret’s ARN instead and fetch it once per execution environment with a TTL cache, or issue a single scoped key pointing at a gateway that holds the real provider credentials.

Conclusion

Serverless and edge functions are a good home for short, bounded, stateless LLM calls and a bad home for long, streaming, stateful ones. The decision reduces to wall-clock time against the smallest ceiling in your request path, and it is worth making per endpoint rather than per system. Where the workload fits: keep the HTTP client at module scope, heartbeat your streams, signal truncation in-band, make retries idempotent, and never put a long-lived provider key in an environment variable. Where it does not: move to a container and stop paying for elasticity you are not using. For the middle ground — bursty front ends, pooled upstream connections, one place to rotate credentials — a gateway such as QoraAPI lets a short-lived function behave like a long-lived client.

Related reading

• Async AI APIs: Job Queues, Webhooks, and Long-Running Tasks
• AI API Streaming Explained: How SSE Works and How to Consume It
• How to Build a Multi-Provider AI Failover Layer for 99.9% Uptime
• How to Build an AI Chatbot with the API

Validate the model’s output because you cannot prove the absence of a failure mode by writing a better instruction. A prompt is a request, not a contract. The same input produces different output across model versions, across providers serving the same weights, and across supposedly identical temperature-0 calls, because batching, kernel selection and floating-point reduction order are not deterministic. Output guardrails are the layer that turns “the model usually does this” into “the system only ever ships this” — and they run after generation, not inside the prompt.

Why you validate the output instead of hardening the prompt

Instruction-following is best-effort. Writing “always return valid JSON” shifts the probability distribution of outputs; it does not create an invariant. That matters because prompt failures are silent: a model that ignores a format instruction returns something plausible until a parser hits it at 3am, whereas a validator fails loudly, at the boundary, with the offending value attached.

Non-determinism has several sources and only one is the temperature parameter. Greedy decoding at temperature 0 breaks ties in the logits, but the logits are a floating-point computation whose reduction order depends on batch size, sequence packing and tensor-parallel layout. Change the batch size and a token ahead by 0.001 logits can flip. Mixture-of-experts routing adds another: a request landing on a different expert set produces a different continuation from the same checkpoint. Version pinning narrows the distribution rather than freezing it, because a pinned snapshot can still be served on different hardware.

So a prompt regression suite tells you what changed, not what is safe right now. Prompts need versioning and review; guardrails are runtime code that needs tests, metrics and an owner. This is about validating what the model says, not what it does — constraining actions is a separate control with a separate threat model, covered in sandboxing AI tool calls.

The validation layers, cheapest first

Run the layers in ascending cost order and stop at the first blocking verdict. A structural failure makes every later check meaningless, and a regex hit is cheaper to act on than a classifier score. Ordering also keeps expensive layers’ error rates out of the picture when a cheap layer already has the answer.

Layer	Added latency	Marginal cost	What it catches	False-positive profile
Structural (parse, JSON Schema)	0.1-2 ms	CPU only	Truncated output, wrong types, missing fields, invented enum values, malformed tool arguments	Near zero if the schema is derived from real payloads; high if the schema is aspirational
Deterministic rules (regex, allow-lists, bounds)	under 1 ms	CPU only	Secret-shaped strings, banned phrases, competitor names, over-length answers, numbers outside a plausible range, tool names not on the allow-list	Entirely a function of how precisely the patterns are written; over-broad regex is the largest source of false positives in most stacks
Groundedness (claim vs retrieved context)	0.3-2 s with a judge; under 50 ms for the prefilter	Cheap for substring and embedding lookups, one model call for the residue	Claims with no support in the context, and claims the context directly contradicts	High when retrieval was truncated or the answer legitimately draws on parametric knowledge; needs a prefilter or it blocks good answers
Classifier-based policy checks	20-150 ms	GPU or hosted classification endpoint, priced per request	Toxicity, system-prompt self-disclosure, injection payloads echoed back, competitor mentions, PII in free text	Dominated by the threshold you pick; you cannot reason about it, you have to measure it on labelled data
Human review (sampled)	Minutes to hours	Highest, by orders of magnitude	Novel failure modes, calibration of the automated layers, cases where two classifiers disagree	Not applicable, but throughput is capped at tens to hundreds of items per reviewer per day

Two properties matter more than the list. Each layer should return findings rather than a boolean, so the final verdict comes from a policy object instead of being hard-coded inside each check. And the false-positive column decides whether a layer ships, not the true-positive column: a detector that catches every real leak but blocks 8 percent of valid traffic is not a detector, it is an outage.

Schema validation in practice

Make the schema strict enough to be worth running

A schema asserting {"type": "object"} catches nothing. The useful constraints are the boring ones: additionalProperties: false so invented fields surface instead of passing downstream, required on every field the consumer dereferences, enum on anything categorical, and maxItems and maxLength to bound payload size.

Provider strict modes are worth enabling — the strict json_schema response format, Anthropic tool-use schemas, Gemini’s responseSchema — but they remove syntax-level failures, not semantic ones, and they do not survive a failover. A strict schema guarantees the shape of the answer, never its truth. Mechanics are in structured outputs and JSON mode.

A repair loop with a hard attempt cap

When validation fails, the cheapest fix is usually to re-ask the same model with the validator’s error attached. That works often, because the failure is frequently a formatting slip rather than a reasoning failure. It needs a cap: every attempt is a full round trip with full input tokens, so two attempts on a 4,000-token prompt is three times the input cost of a single call.

import json
from dataclasses import dataclass
from typing import Any, Callable

import jsonschema

INVOICE_SCHEMA: dict[str, Any] = {
    "type": "object",
    "additionalProperties": False,
    "required": ["vendor", "currency", "total_cents", "line_items"],
    "properties": {
        "vendor": {"type": "string", "minLength": 1, "maxLength": 200},
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
        "total_cents": {"type": "integer", "minimum": 0},
        "line_items": {
            "type": "array",
            "minItems": 1,
            "maxItems": 200,
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["description", "amount_cents"],
                "properties": {
                    "description": {"type": "string", "maxLength": 500},
                    "amount_cents": {"type": "integer", "minimum": 0},
                },
            },
        },
    },
}

MAX_REPAIR_ATTEMPTS = 2

@dataclass(frozen=True)
class SchemaResult:
    ok: bool
    value: dict[str, Any] | None
    attempts: int
    errors: list[str]

def _describe(exc: Exception) -> str:
    if isinstance(exc, jsonschema.ValidationError):
        path = "/".join(str(p) for p in exc.absolute_path) or "<root>"
        return f"{path}: {exc.message}"
    return str(exc)

def parse_with_repair(
    raw: str,
    call_model: Callable[[str, float], str],
    temperature: float = 0.0,
) -> SchemaResult:
    """Validate raw output; on failure re-ask with the validator error attached.

    call_model is injected so this is unit-testable without a network call.
    """
    errors: list[str] = []
    candidate = raw
    for attempt in range(MAX_REPAIR_ATTEMPTS + 1):
        try:
            value = json.loads(candidate)
            jsonschema.validate(value, INVOICE_SCHEMA)
        except (json.JSONDecodeError, jsonschema.ValidationError) as exc:
            errors.append(_describe(exc))
        else:
            return SchemaResult(True, value, attempt, [])

        if attempt == MAX_REPAIR_ATTEMPTS:
            break

        candidate = call_model(
            "Your previous reply failed validation. Fix only the problems listed. "
            "Do not add, remove or reinterpret fields, and do not change a value "
            "that was not listed as invalid.\n"
            "Errors:\n" + "\n".join(f"- {e}" for e in errors[-3:]) + "\n"
            "Return the corrected JSON object and nothing else.",
            temperature,
        )

    return SchemaResult(False, None, MAX_REPAIR_ATTEMPTS, errors)

What the repair loop must never do

The repair prompt above carries a sentence doing real work: do not change a value that was not listed as invalid. Without it, a model asked to fix a schema error will often rewrite the data to make the error disappear — rounding a total so the line items sum, deleting the field with the wrong type, reclassifying a currency to fit the enum. That is data corruption dressed as a successful repair.

Cross-field semantic checks belong outside the repair loop. If your validator asserts that sum(line_items) == total_cents, a repair cannot fix a mismatch, because the model cannot know which side is wrong. Block and route to a human. A repair rate jumping from 2 percent to 15 percent after a prompt edit is the earliest signal of a regression.

Groundedness: unsupported is not the same as contradicted

Extract the atomic claims, then look for support for each in the context you actually retrieved. Three outcomes, and the third is the important one. Supported: a span in the context entails the claim. Unsupported: nothing speaks to it either way — the model may be using parametric knowledge, or retrieval was truncated. Contradicted: the context asserts the opposite. Contradicted claims should be blocked or rewritten; unsupported claims should be downgraded or shown with a citation requirement, because blocking every unsupported claim destroys a summarizer that adds one reasonable inference. See detecting and reducing hallucinations for the taxonomy in depth.

Per-claim judging is quadratic in the obvious implementation, so add a prefilter: normalize each claim, then check for a near-verbatim span, an embedding similarity above a tuned threshold, or a token-overlap ratio. Only the residue goes to a judge, and the judge call is batched — one request containing every unresolved claim, not one per claim.

Worked example. A response contains 40 atomic claims. The prefilter resolves 28 of them, or 70 percent, without a model call. The remaining 12 go to a judge in one batched request returning per-claim entailment labels in roughly 400 ms. Added latency is one round trip, not twelve. Calling the judge once per unresolved claim at 350 ms each would add 4.2 seconds, and the feature would be disabled within a week.

If you use a judge, use a different model than the generator, or at minimum a different prompt with a strict rubric. Ask for a span-level verdict — “quote the sentence that supports this claim, or answer NONE” — rather than a scalar score. A judge forced to produce a citation cannot reward itself with a vague 0.8.

PII and secret leakage in the output

Deterministic detectors cover more than people expect: card numbers with a Luhn check, IBANs with mod-97, national identifier formats, AWS access key prefixes, JWTs by their three-segment structure, PEM private key blocks, connection strings with embedded credentials. Use the checksum wherever the format has one, because a bare digit pattern also matches order numbers.

Regex alone misses the cases that matter: a paraphrased address, a phone number written in words, a name split across a token boundary, an email base64-encoded into a code block, “the card ending in 4242”, or an identifier the model reconstructed from context rather than copied. For free-text names, addresses and locations you need a NER or classifier pass in addition, and it will be the layer with the least predictable error rate.

Redaction versus blocking is a decision about reversibility, not severity. Mask an email as a stable placeholder when the consumer only needs to know an email was present, and keep the mapping server-side. Block when the value is a credential, a regulated identifier, or anything whose exposure is itself the harm. And never log the raw pre-guardrail output: if your guardrail redacts PII and your observability pipeline then stores the original response body, the guardrail has achieved nothing except latency. Log the post-guardrail text plus finding metadata — layer, rule, span offsets, verdict — and put raw output, if you must keep it, in a separate store with short retention and its own audit trail. The wider constraints are in AI data privacy and GDPR.

Fail-open or fail-closed, decided per surface

Fail-open means that when the guardrail itself errors or times out, the output ships. Fail-closed means it does not. The answer is not global, and it is not something you discover in an except block — it is configuration attached to the surface.

Blast radius decides. A support chatbot failing open ships one bad answer to one user who can ask again; the damage is bounded. The same chatbot failing closed turns every classifier timeout into “something went wrong” for the whole user base, an availability incident you caused yourself. A clinical triage flow inverts the calculus: failing open can produce advice that causes physical harm, while failing closed sends the user to a phone number a human answers. Internal analytics fails open by default, because a wrong number in a dashboard is cheap to correct.

Be precise about timeouts. A classifier that times out has said nothing about the content; treating that as “unsafe” produces random outages under load, and load is exactly when timeouts cluster.

Latency budgeting and the streaming problem

Guardrails add round trips, and the budget is tighter than teams assume. Run deterministic checks inline; they cost microseconds. Run the classifier and the groundedness check concurrently, because they are independent, so the wall-clock cost is the maximum rather than the sum.

Worked example. A response is 600 tokens. The PII detector takes 15 ms, the toxicity classifier 90 ms, the policy classifier 120 ms and the groundedness judge 400 ms. Serially that is 625 ms of added latency before the first token reaches the client. With asyncio.gather over the last three it is max(90, 120, 400) = 400 ms. The judge is the critical path, which is why the groundedness prefilter buys more than any micro-optimization of the regexes.

Hold-back windows and retraction UX

Streaming and blocking are in direct tension. Once a token is on the user’s screen you cannot un-show it; you can only append a retraction, and you should assume the user read the original. Three patterns work. First-N-token gating buffers the opening 40 to 80 tokens, validates it, and releases the stream if it passes; most violations are visible in the opening. A sliding hold-back window keeps the last W tokens unreleased while checks run, bounding exposure to the window. Full buffering generates, validates, then renders, which for a 600-token answer at 60 tokens per second moves time-to-first-token from about 0.5 s to about 10 s.

type GateResult = { block: boolean; reason?: string };

export type Gate = {
  holdbackChars: number;
  check: (unreleased: string) => Promise<GateResult>;
};

export async function* guardedStream(
  source: AsyncIterable<string>,
  gate: Gate,
): AsyncGenerator<string> {
  let pending = "";
  let released = 0;

  for await (const chunk of source) {
    pending += chunk;

    // Only inspect text that has not reached the client yet.
    const unreleased = pending.slice(released);
    if (unreleased.length < gate.holdbackChars) continue;

    const { block, reason } = await gate.check(unreleased);
    if (block) {
      yield `\n\n[Response withheld: ${reason ?? "policy"}]`;
      return;
    }

    yield unreleased;
    released = pending.length;
  }

  const tail = pending.slice(released);
  if (tail) {
    const { block, reason } = await gate.check(tail);
    yield block ? `\n\n[Response withheld: ${reason ?? "policy"}]` : tail;
  }
}

The window bounds the text a user can see before a violation is caught, so a retraction undoes a sentence rather than a paragraph. It does not help when the violation is a single token in the middle of a long answer. For high-harm surfaces, accept the latency and buffer the whole response.

Fallback strategies

A blocked response is a routing decision, not an error state. Choose the fallback before you ship the guardrail, because inventing one during an incident produces worse outcomes than a boring canned string.

Fallback	Use when	Added latency	Cost per event	What the user sees
Safe canned response	The block was for an out-of-scope or off-policy request	Under 10 ms	Effectively zero	A short answer that declines and points to docs or a human
Degrade to a smaller model with a stricter prompt	The violation looks prompt-shaped and a constrained model is good enough for the task	One extra generation, 300 ms to 2 s	One additional generation plus the blocked one	A slightly less rich but valid answer
Hand off to a human	The domain is high-harm and the request is legitimate but unresolvable automatically	Minutes	Highest, and unbounded per item	A queue position, a callback promise or a ticket number
Structured error	The caller is a machine, not a person	Under 10 ms	Effectively zero	A typed error with a retryable flag and the failing rule id

Two rules of thumb. Degrade to a smaller model when the failure came from a permissive prompt, and not when the task is simply hard — a weaker model on a hard task produces a confident wrong answer, which is worse than a block. And never let a fallback re-enter the same guardrail loop without a depth limit, or a persistent violation becomes an infinite regeneration loop.

Measuring a guardrail you intend to ship

Build a labelled set before shipping the layer: a sample of real traffic, human-labelled at the exact verdict granularity you ship, plus a synthetic adversarial set covering the failures you worry about. Then track precision, recall and the false-positive rate on live traffic, split by layer, because a blended number hides which layer is misfiring.

Worked example. You serve 1,000,000 responses a month and the blocking guardrail fires on 0.5 percent of them, so 5,000 blocks. If the measured false-positive rate on known-good outputs is 5 percent, then roughly 0.05 x 995,000 = 49,750 legitimate responses were blocked. You have manufactured an incident an order of magnitude larger than the one you were preventing. You cannot measure that 5 percent from blocked traffic alone — it requires labelling outputs you allowed, which is the step teams skip.

Two more metrics belong on the same dashboard: repair-loop rate, a leading indicator of prompt or model regression, and block rate per rule, because a rule whose block rate is zero is either perfect or broken and it is almost never perfect. Re-run the labelled set on every model version change. Keeping model and prompt changes traceable alongside these numbers is where LLM observability earns its keep. A guardrail nobody measured is one you cannot ship.

A concrete pipeline

The shape that survives production is a set of layer functions returning findings, a policy object resolved per surface, and a verdict computed by severity rather than control flow. The classifier is injected, so the whole thing is testable with a stub.

import asyncio
import re
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Awaitable, Callable

class Verdict(str, Enum):
    ALLOW = "allow"
    REDACT = "redact"
    BLOCK = "block"
    REVIEW = "review"

@dataclass
class Finding:
    layer: str
    rule: str
    verdict: Verdict
    detail: str

@dataclass
class Outcome:
    verdict: Verdict
    text: str
    findings: list[Finding] = field(default_factory=list)
    timings_ms: dict[str, float] = field(default_factory=dict)

SECRET_PATTERNS = {
    "aws_access_key": re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b"),
    "private_key_block": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    "jwt": re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b"),
}

Policy = dict[str, Verdict]

# One explicit policy per surface. Fail-open and fail-closed are data here,
# not an except block somewhere in the request handler.
POLICIES: dict[str, Policy] = {
    "support_chat": {
        "structural": Verdict.BLOCK, "policy": Verdict.BLOCK,
        "grounded": Verdict.REDACT, "on_error": Verdict.ALLOW,
    },
    "clinical_triage": {
        "structural": Verdict.BLOCK, "policy": Verdict.BLOCK,
        "grounded": Verdict.BLOCK, "on_error": Verdict.BLOCK,
    },
    "internal_analytics": {
        "structural": Verdict.REVIEW, "policy": Verdict.REVIEW,
        "grounded": Verdict.ALLOW, "on_error": Verdict.ALLOW,
    },
}

WORST_FIRST = (Verdict.BLOCK, Verdict.REDACT, Verdict.REVIEW, Verdict.ALLOW)

def _worst(findings: list[Finding], fallback: Verdict) -> Verdict:
    for verdict in WORST_FIRST:
        if any(f.verdict is verdict for f in findings):
            return verdict
    return fallback

def redact(text: str, findings: list[Finding]) -> str:
    out = text
    for f in findings:
        if f.layer == "pii" and f.detail:
            out = out.replace(f.detail, f"[REDACTED:{f.rule}]")
    return out

async def run_guardrails(
    text: str,
    surface: str,
    *,
    validate_structure: Callable[[str], list[Finding]],
    classify: Callable[[str], Awaitable[list[Finding]]],
    check_groundedness: Callable[[str], Awaitable[list[Finding]]],
) -> Outcome:
    policy = POLICIES[surface]
    timings: dict[str, float] = {}

    start = time.perf_counter()
    structural = validate_structure(text)
    timings["structural"] = (time.perf_counter() - start) * 1000
    if structural:
        # A structural failure makes every later layer meaningless.
        return Outcome(_worst(structural, policy["structural"]), text, structural, timings)

    start = time.perf_counter()
    rule_findings = [
        Finding("rules", name, Verdict.BLOCK, "secret-shaped string in output")
        for name, pattern in SECRET_PATTERNS.items()
        if pattern.search(text)
    ]
    timings["rules"] = (time.perf_counter() - start) * 1000

    start = time.perf_counter()
    policy_findings, grounded_findings = await asyncio.gather(
        classify(text), check_groundedness(text)
    )
    timings["classifier_and_grounded"] = (time.perf_counter() - start) * 1000

    findings = rule_findings + policy_findings + grounded_findings
    verdict = _worst(findings, Verdict.ALLOW)
    if verdict is Verdict.REDACT:
        text = redact(text, findings)

    return Outcome(verdict, text, findings, timings)

Three choices in that code are load-bearing. Structural failures short-circuit, because there is nothing useful to say about the policy of unparseable output. Classifier and groundedness run under one gather, so added latency tracks the slower of the two. And the error path lives in POLICIES rather than a bare except, so an operator can change it without reading the request handler.

What I would ship first for a customer-facing product

Ship structural validation with a bounded repair loop first, before any classifier. It is free, its false-positive rate is near zero when the schema comes from real payloads, and it eliminates the failures that page you at 3am: truncated JSON, missing fields, wrong types. Instrument the repair rate from day one.

Second, ship deterministic PII and secret detection, fail-closed on secrets and redact on everything else, with raw output kept out of logs from the first commit. It is cheap, defensible in a security review, and the layer most likely to catch a genuine incident.

Third, buffer the full response for the single highest-harm surface, and only that surface. Non-streaming is a real product cost, so spend it where the blast radius justifies it.

Do not ship a policy classifier as a blocker before you have a labelled set and a measured false-positive rate. Do not ship an LLM-as-judge groundedness check as a blocker at all in the first release: run it in shadow mode, log what it would have blocked, and label a few hundred of those decisions by hand. Until you can state your false-positive rate with a number attached, it is a measurement, not a control.

Put model routing behind a gateway rather than wiring providers into application code, so pinning a version, swapping a judge or failing over is a configuration change instead of a deploy.

Frequently asked questions

Do I still need output validation if I use a provider’s strict structured outputs mode?

Yes, for three reasons. Strict modes constrain syntax, not semantics: a strict schema will happily return a well-typed total that does not equal the sum of its line items. They are provider-specific, so the guarantee disappears the moment your gateway fails over. And they say nothing about groundedness, PII or policy.

Should the groundedness judge be a different model from the generator?

Preferably yes, and at minimum it must be a different prompt with an explicit rubric. A judge sharing the generator’s system prompt inherits the same framing and will rationalise the generator’s output. If you can only afford one model, demand a quoted supporting span for every supported verdict.

Can I stream tokens and still enforce guardrails?

Partially. Use a hold-back window so the text you validate has not reached the client, and accept that a late violation still needs a retraction. First-N-token gating covers violations in the opening sentence. Where a leaked token is unacceptable, do not stream.

How do I handle a guardrail false positive in production?

Make it reviewable rather than just blockable. Store the finding, the rule id, the span and a hash of the output, so a reviewer can judge the decision without storing raw text, and give the reviewer an override that feeds back into the labelled set. If one rule accounts for most of your false positives, fix that rule rather than raising the layer threshold.

Do guardrails belong in the client, the gateway, or the application?

Split them by what each knows. The application owns semantic checks, because only it knows the schema, the retrieved context and the policy per surface. The gateway owns the cross-cutting pieces: version pinning, provider failover, redacted logging, rate limits. The client should own nothing that matters for safety.

Conclusion

Output guardrails are the executable specification of what your system is allowed to emit. Order them by cost, make each layer return findings rather than booleans, resolve fail-open versus fail-closed per surface as configuration, and measure the false-positive rate before anything is allowed to block. The layers that ship first are the boring ones: schema validation with a bounded repair loop, deterministic secret and PII detection, and full buffering where a leaked token is unacceptable.

The alternative is not “no guardrails”, it is unmeasured guardrails: regexes nobody tuned, a threshold copied from a blog post, and no idea how many good answers you blocked last month. Routing model traffic through a single OpenAI-compatible endpoint such as qoraapi.com makes the operational side tractable — pinned versions, failover, one place to enforce redaction and logging — but the validation logic is still yours to design.

Related reading

• Detecting and Reducing Hallucinations in Production LLM Apps
• AI Structured Outputs Explained: JSON Mode, Schema Enforcement, Reliable Parsing
• Sandboxing AI Tool Calls: Preventing Data Exfiltration
• AI API Security: Protecting Keys and Preventing Abuse

An LLM eval harness is a versioned set of task-specific cases plus scorers, run on every change to your prompt, model or retrieval stack, that answers one question: did this change break something a user would notice? It is not a leaderboard. MMLU and arena rankings say nothing about whether your invoice extractor still returns the right currency for a scanned PDF from a German vendor.

This is the application-level view: scoring your feature, not the model. Public benchmarking is covered in how to evaluate and benchmark AI models; throughput rather than correctness is load testing.

Why public benchmarks do not predict your behaviour

Three reasons, all structural.

Distribution mismatch. Benchmarks sample curated, clean questions. Your traffic is messy: half-paragraph tickets, OCR noise, three languages in one message, a stack trace pasted into a description box. That gap is not a constant you can subtract.

Benchmarks measure the model alone. Your feature is a prompt, retrieved documents, a tool-call loop, a parser and a retry policy. A model that gains points on a reasoning benchmark can make your feature worse, because it grew more verbose and your parser truncates at 2,000 tokens.

Published benchmarks get optimised against. Widely reported numbers become training targets, which makes them non-transferable.

Building the golden set

Source cases from traffic and incidents

The best cases are not invented. They come from logs and postmortems.

Sample real inputs, but stratify rather than taking the top hundred by frequency: head traffic already works. Take the long tail — shortest input, longest input, emoji in the middle, two attachments with a question about the second. Where a router sits upstream, sample branches evenly rather than by traffic share.

Every user-visible incident becomes a permanent case: a ticket saying “the summary was about the wrong account” enters the set with an assertion that would have caught it. Never invent cases in a text editor, because you will unconsciously write the ones your prompt already passes.

Thirty to a hundred cases to start

Below 30 cases almost nothing is detectable: moving three of them swings the score by ten points. Above roughly 150, marginal information per case falls fast, because you start adding near-duplicates. The first 50 capture the failure modes you know; the rest buy the second and third standard deviations.

Stratify, weight, and hold something back

Tag every case on the axes that matter: difficulty, user segment, input shape and risk — the difference between an answer that is annoying and one that is a compliance problem. A pass rate moving from 91 to 89 percent is a number; a breakdown showing the drop is entirely in the long-input bucket, paid tier only, is a diagnosis.

Weight by risk tag, because a wrong currency code costs more than a verbose sentence. Keep a held-out slice of 20 to 30 percent, picked by a fixed seed and treated as read-only: edit a prompt in response to a failure there and it stops being held out.

Case anatomy

• Input — verbatim. Preserve whitespace and typos; normalising them changes what you test.
• Expected properties — a dict of assertions, not a string. “Names the vendor and the total is within one cent of 4210.50” is a property; “is exactly this sentence” is not.
• Context and fixtures — retrieved documents, tool responses, a frozen row. Pin them: an eval that hits a live index turns an index change into an apparent model regression.
• Tags — the stratification axes above, as a flat tuple you can group by.
• Weight or risk — what a failure here costs, so the aggregate means something.

Exact match is usually the wrong assertion

Generative output has many correct realisations, so asserting the exact string fails on every model upgrade, temperature change and tokenizer whitespace difference. Those failures are noise, and teams end up muting the suite.

Assert properties that must hold for any acceptable answer. Summarisation: names all three required entities, under 120 words, no number absent from the source, no competitor mentioned. Classification: the label is in the allowed set. Extraction: the JSON parses, the schema validates, the amount is within tolerance. Exact match survives only where the output space is a small closed enum.

Choosing a scorer

Match the scorer to the failure you want to detect, not to the one you find easiest to write.

Scorer	Use it when	Failure mode	Cost per case
Exact or regex assertion	The output space is closed: labels, enums, IDs, fixed-format codes	Brittle to any legitimate variation; passes when a wrong answer happens to contain the pattern	CPU, microseconds
Programmatic check (parses, schema valid, citation resolves, number in tolerance)	Output is structured, or a property is mechanically checkable	Catches only what you encoded; a valid-but-wrong answer passes	CPU plus any lookup latency
Embedding similarity to a reference answer	A reference exists and paraphrases are acceptable, as in semantic search or FAQ answers	High similarity for answers that are confidently wrong on the same topic; the threshold is dataset-specific and needs calibration	One embedding call, cacheable
Model-graded with a rubric (LLM-as-judge)	The criterion is subjective or needs reasoning: groundedness, tone, whether a refusal was appropriate	Position, verbosity and self-preference bias; judge noise becomes suite noise	One or more extra generations, often the dominant cost
Human review, sampled	Calibrating the judge, adjudicating disagreements, triaging novel failures	Throughput; drift in reviewer standards over weeks	Minutes of human time

Programmatic checks earn more than they look like

The strongest assertions in production suites are mechanical: the response parses as JSON; required fields are non-empty; every citation ID exists in the retrieved set; every number in the output appears in the context or a tool result; no write tool call precedes a confirmation. These are deterministic, free and fast, and they catch most real regressions. Substring grounding is a crude proxy for hallucination but catches a lot before you pay for a judge — see detecting and reducing hallucinations. Layer the judge on top for what no rule expresses: was the refusal appropriate, did the answer address the question, is the tone within policy.

LLM judges have three biases you must design around

Position bias. Comparing two answers, the judge prefers whichever came first, or last, more often than chance. Run each pair twice with the order swapped, count the comparison only when both runs agree, record disagreements as ties. That doubles judge cost and is worth it.

Verbosity bias. Longer answers score higher even when the extra content is filler. Put an explicit length constraint in the rubric and, where length is irrelevant, state that longer is not better. Better still, measure length programmatically.

Self-preference. A model grading its own output scores it higher than a different model would. Use a judge from a different model family than the one under test, and do not silently swap the judge when you swap the model. Prefer pairwise comparison to absolute scoring for subjective criteria: “is A better than B” is more stable than “rate this one to seven”, and absolute scales drift between judge versions.

Writing a rubric a judge can apply consistently

A rubric is code. If it is ambiguous it produces inconsistent verdicts, and that inconsistency surfaces as flakiness in your suite.

Define the boundary of pass explicitly. Not “the answer should be accurate”. Write: pass if every factual claim is supported by the provided context; fail if any claim is unsupported or contradicted. Then give one worked boundary case, such as a claim that is true but absent from the context, and state the verdict.

Use few discrete levels and name the middle one. A three-level scale of fail, partial and pass with an explicit definition of partial is easier to apply consistently than a one-to-ten scale. Define partial as a concrete condition: “supported and correct but omits one of the three required entities”. Treat partial as a fail for gating and report it separately: a partial that silently counts as 0.5 makes the pass rate hard to reason about. Require the judge to return structured JSON with a mandatory rationale, which is what lets a human audit it when a case flips.

Running evals in CI

The core tension: a full suite with a judge costs money and minutes, and developers route around anything slow. Split the suite.

On every pull request, run 20 to 40 cases covering each tag bucket, scored deterministically plus a judge on the highest-risk tags only: under three minutes and a few cents. Nightly, run the full suite including the held-out slice, storing results with the model fingerprint and prompt hash attached.

Cache model responses. Key the cache on a hash of model identifier, prompt version hash, temperature, decoded input and fixture versions. If none changed, the recorded output is a valid replay, which turns most PR runs into pure scorer runs.

Pin model versions. Never point the suite at a floating alias. Record the resolved model identifier in every result row and treat a change in it as a re-baseline event rather than a regression, because providers do update snapshots behind stable names. Routing every candidate through one OpenAI-compatible endpoint makes pinning and swapping tractable, which is one reason to keep eval traffic on a gateway such as Qora.

A single run cannot gate a merge

Evals are stochastic: even at temperature zero, provider-side batching and hardware variation produce different outputs from identical inputs, so a one-case drop between two runs of the same commit is not evidence of anything. Gate on the deterministic scorers, which are stable: block the merge if a deterministic assertion fails on a case it previously passed, or if the pass rate falls by more than the interval width, and warn rather than block on smaller judge-based movement.

The statistics that actually matter

How many cases before a difference is meaningful. The margin of error for a proportion is roughly one over the square root of n at 95 percent confidence, worst case near p = 0.5. So 100 cases give about plus or minus 10 points, 400 give about 5, and 1,600 give about 2.5. At 40 cases you are near plus or minus 15 points: a 40-case suite detects breakage and not much else, which is what pages you.

Report an interval, always. “89 percent” invites a comparison with “91 percent” that 100 cases cannot support. “89 percent, 95 percent CI [81, 94]” makes the comparison honest at a glance. For small n and rates near zero or one, use the Wilson interval rather than the normal approximation, which produces bounds below zero and behaves badly exactly where eval suites live.

Flakiness versus regression. Run the same code twice. A case that fails in both runs is a regression candidate. A case that fails once and passes once is flaky, either because behaviour is genuinely nondeterministic near a boundary or because the judge cannot apply the rubric consistently. Quarantine flaky cases with an owner rather than deleting them.

What the harness itself costs

Work the arithmetic before building: the judge is the line item that surprises people.

Take 100 cases, each with roughly 800 input tokens of prompt, fixture and question, producing roughly 300 output tokens. A full run is 80,000 input and 30,000 output tokens. Add a judge at 1,200 input tokens per case — rubric plus context plus answer — and 150 output tokens: another 120,000 input and 15,000 output. Call it 200,000 input and 45,000 output per full run.

At a blended $3 per million input tokens and $15 per million output tokens, that is $0.60 plus $0.68, so about $1.28 per run and roughly $39 a month nightly. Add a 40-case PR subset across 60 pull requests a month with a judge on half of them: 60 x 20 x 1,200 equals 1.44 million judge input tokens, about $4.32, plus a few dollars of output. The whole harness lands under $60 a month before caching, and caching pushes it far below, because a PR run only pays for cases whose inputs, fixtures or prompt actually changed.

Latency follows the same shape. Judge calls take one to three seconds each, so a judge on every case dominates wall time; deterministic scorers first is what keeps a PR run short.

Drift: three things that change while you change nothing

Eval suites fail in a specific way when unmaintained: they keep passing while the product gets worse.

A provider silently updates a model. A snapshot alias keeps its name and changes its weights or serving stack. Record the resolved model identifier and, where the provider exposes one, a system fingerprint in every result row; a change in either is a re-baseline event, not a regression. Pinning to a dated snapshot reduces the frequency without eliminating it, since the same snapshot can be served on different hardware.

Your prompt file changes. Hash the fully rendered prompt after variable substitution, after tool definitions are serialised and after few-shot examples are included, not the template file on disk. Teams have shipped regressions because the hash covered the template while a fixture change altered the rendered text. Rollout practice is covered in prompt management and versioning.

Your retrieval index changes. A re-embedding job, a chunking tweak or a document deletion changes what the model sees without touching the prompt. Freeze retrieved context as a fixture so the suite tests generation deterministically, and run a separate retrieval eval measuring whether the expected document is recalled for each query: one eval cannot measure both, or you cannot attribute a failure. Traces carrying retrieved document IDs let you reconstruct which index state produced a bad answer, which is the job of LLM observability.

A result is only meaningful when you can name every input to the system — model identifier, prompt hash, fixture version, index version, scorer version. If any is missing from the row, the number is not comparable to next week’s.

A harness skeleton

Four pieces: a case, a scorer protocol, a runner with a concurrency cap and a disk cache, and a report with intervals and a per-tag breakdown.

from __future__ import annotations

import asyncio
import hashlib
import json
import math
import re
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Awaitable, Callable, Protocol, Sequence

@dataclass(frozen=True)
class Case:
    id: str
    input: str
    expect: dict[str, Any] = field(default_factory=dict)
    context: list[str] = field(default_factory=list)
    tags: tuple[str, ...] = ()
    weight: float = 1.0
    split: str = "tune"          # "tune" | "holdout"

@dataclass(frozen=True)
class Verdict:
    passed: bool
    score: float                 # 0.0 .. 1.0, so partial credit stays visible
    detail: str = ""

class Scorer(Protocol):
    name: str
    async def __call__(self, case: Case, output: str) -> Verdict: ...

class JsonSchemaScorer:
    name = "json_schema"

    def __init__(self, required: Sequence[str]) -> None:
        self.required = required

    async def __call__(self, case: Case, output: str) -> Verdict:
        try:
            payload = json.loads(output)
        except json.JSONDecodeError as exc:
            return Verdict(False, 0.0, "not json: " + exc.msg)
        if not isinstance(payload, dict):
            return Verdict(False, 0.0, "top level is not an object")
        missing = [k for k in self.required if not payload.get(k)]
        if missing:
            return Verdict(False, 0.0, "missing: " + ",".join(missing))
        return Verdict(True, 1.0, "")

class GroundedNumbersScorer:
    name = "grounded_numbers"
    pattern = re.compile(r"\d+(?:[.,]\d+)?")

    async def __call__(self, case: Case, output: str) -> Verdict:
        haystack = " ".join(case.context)
        stray = [n for n in self.pattern.findall(output) if n not in haystack]
        if stray:
            return Verdict(False, 0.0, "unsupported numbers: " + str(stray[:5]))
        return Verdict(True, 1.0, "")

def cache_key(model: str, prompt_hash: str, case: Case, seed: int) -> str:
    blob = json.dumps(
        {
            "model": model,
            "prompt": prompt_hash,
            "input": case.input,
            "context": case.context,
            "seed": seed,
        },
        sort_keys=True,
    )
    return hashlib.sha256(blob.encode()).hexdigest()

class Cache:
    def __init__(self, path: Path) -> None:
        self.path = path
        self.path.mkdir(parents=True, exist_ok=True)

    def get(self, key: str) -> str | None:
        f = self.path / (key + ".txt")
        return f.read_text(encoding="utf-8") if f.exists() else None

    def put(self, key: str, value: str) -> None:
        (self.path / (key + ".txt")).write_text(value, encoding="utf-8")

@dataclass
class Result:
    case: Case
    output: str
    verdicts: list[Verdict]

    @property
    def passed(self) -> bool:
        return all(v.passed for v in self.verdicts)

async def run_case(
    case: Case,
    generate: Callable[[Case], Awaitable[str]],
    scorers: Sequence[Scorer],
    cache: Cache,
    model: str,
    prompt_hash: str,
    seed: int,
) -> Result:
    key = cache_key(model, prompt_hash, case, seed)
    output = cache.get(key)
    if output is None:
        output = await generate(case)
        cache.put(key, output)
    verdicts = [await s(case, output) for s in scorers]
    return Result(case, output, verdicts)

async def run_suite(
    cases: Sequence[Case],
    generate: Callable[[Case], Awaitable[str]],
    scorers: Sequence[Scorer],
    cache: Cache,
    model: str,
    prompt_hash: str,
    concurrency: int = 12,
    seed: int = 0,
) -> list[Result]:
    gate = asyncio.Semaphore(concurrency)

    async def one(case: Case) -> Result:
        async with gate:
            return await run_case(
                case, generate, scorers, cache, model, prompt_hash, seed
            )

    return await asyncio.gather(*(one(c) for c in cases))

def wilson(passed: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    p = passed / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (max(0.0, centre - half), min(1.0, centre + half))

def report(results: Sequence[Result], split: str = "tune") -> dict[str, Any]:
    rows = [r for r in results if r.case.split == split]
    if not rows:
        return {"split": split, "n": 0}

    passed = sum(1 for r in rows if r.passed)
    n = len(rows)
    lo, hi = wilson(passed, n)

    by_tag: dict[str, list[bool]] = {}
    for r in rows:
        for tag in r.case.tags:
            by_tag.setdefault(tag, []).append(r.passed)

    weight_total = sum(r.case.weight for r in rows)
    weight_pass = sum(r.case.weight for r in rows if r.passed)

    return {
        "split": split,
        "n": n,
        "pass_rate": round(passed / n, 4),
        "ci95": [round(lo, 4), round(hi, 4)],
        "weighted_pass_rate": round(weight_pass / weight_total, 4),
        "by_tag": {
            tag: {
                "n": len(v),
                "pass_rate": round(sum(v) / len(v), 4),
                "ci95": [round(x, 4) for x in wilson(sum(v), len(v))],
            }
            for tag, v in sorted(by_tag.items())
        },
    }

Two details are load-bearing. The cache key includes the prompt hash and the model identifier, so a prompt edit or model swap invalidates exactly the affected rows. And the report emits an interval next to every rate, because tag-level numbers come from the smallest samples.

A CI configuration

A PR subset that blocks on deterministic failures, and a nightly full run across both splits.

name: llm-evals

on:
  pull_request:
  schedule:
    - cron: "0 3 * * *"        # nightly full run, 03:00 UTC

jobs:
  pr-subset:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r evals/requirements.txt
      - name: Run PR subset
        env:
          QORA_API_KEY: ${{ secrets.QORA_API_KEY }}
        run: |
          python -m evals.run \
            --suite suites/support-agent.yaml \
            --split tune \
            --subset-per-tag 4 \
            --scorers json_schema,grounded_numbers,citation_exists \
            --judge-tags risk:high \
            --judge-samples 1 \
            --cache .evalcache \
            --report pr-report.json \
            --fail-on-deterministic \
            --max-pass-rate-drop 0.05
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pr-report
          path: pr-report.json

  nightly-full:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    timeout-minutes: 45
    strategy:
      matrix:
        split: [tune, holdout]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r evals/requirements.txt
      - name: Run full suite
        env:
          QORA_API_KEY: ${{ secrets.QORA_API_KEY }}
        run: |
          python -m evals.run \
            --suite suites/support-agent.yaml \
            --split ${{ matrix.split }} \
            --scorers json_schema,grounded_numbers,citation_exists,judge \
            --judge-model claude-sonnet-4-5 \
            --judge-samples 3 \
            --runs 3 \
            --concurrency 16 \
            --cache .evalcache \
            --record-model-fingerprint \
            --baseline baselines/${{ matrix.split }}.json \
            --report full-${{ matrix.split }}.json
      - uses: actions/upload-artifact@v4
        with:
          name: full-report-${{ matrix.split }}
          path: full-${{ matrix.split }}.json

The --runs 3 flag separates flakiness from regression: mark a case failed only if it fails in every run. The --baseline flag compares against the last accepted full run rather than an absolute threshold, which is the only comparison that survives normal drift.

What to build in week one

Week one: forty cases from the last two weeks of logs and the last ten incidents, stored as YAML in the repository. Deterministic scorers only: schema, required fields, citation existence, number grounding, length. A runner of about a hundred lines that writes a JSON report with Wilson intervals, plus disk caching. A CI job on every PR that blocks on deterministic failures only.

Weeks two to four: add the judge with a written rubric for the two or three criteria you cannot express mechanically, and calibrate it against thirty human-labelled outputs before you trust it. Add the nightly full run and the held-out split, and start recording model fingerprints.

What can wait: a web dashboard, a results database, an annotation UI, pairwise model comparison, automatic prompt optimisation, multi-turn evals, and fine-tuning the judge. None find a regression in week one. The most common failure mode is building the framework instead of the cases: a hundred hand-written cases with a sixty-line runner beats an elegant plugin architecture holding eight placeholders.

Frequently asked questions

How is an eval harness different from unit testing?

Unit tests assert on deterministic functions; an eval harness asserts on a distribution. A single failure may be sampling noise rather than a bug, so you report rates with intervals instead of a binary build status, and the expected value is usually a set of properties. Keep ordinary unit tests for the parsing, routing and prompt-rendering code around the model: those are exact, fast, and catch regressions that never needed a model call.

Should I use an off-the-shelf eval framework?

Use one if it fits, but keep the case format and the scorers yours. The case file and scorer implementations encode what your product must do; the runner, caching and reporting are commodity. Teams that adopt a framework’s case format wholesale often find it cannot express their assertions.

How often should I update the golden set?

Add cases continuously from incidents: every user-visible failure gets one. Prune rarely, and only when a case duplicates another and covers no distinct tag combination. Never delete a case because it fails, and review the set quarterly for cases that assert an implementation detail rather than a product property.

Do I need a judge if my outputs are structured?

No. If every output is a JSON object validated against a strict schema and every field is checkable against the input, deterministic scorers cover you. Judges earn their place when the output is prose, when the criterion is subjective such as tone or the appropriateness of a refusal, or when correctness needs reasoning over the context.

How do I evaluate a multi-turn agent?

Make the trajectory the unit of assertion, not the final message. Record tool calls with their arguments and assert on the sequence: no write before a confirmation, no more than N tool calls, the final answer cites a tool result. For the judge, feed the whole trajectory and the tool results, not just the last message.

Conclusion

An application eval harness is a small amount of machinery around a hard part: deciding what “still works” means for your feature, in a form a machine can check. Cases come from production, assertions are properties rather than strings, scorers run cheapest first, and results are reported with confidence intervals rather than a single number.

Two disciplines hold it together. Pin and record everything that feeds the system, so a result is comparable to the previous one. And treat the harness as production code with an owner, because an unmaintained suite drifts into passing while the product regresses, which is worse than having no suite at all.

Related reading

• Evaluating and Benchmarking AI Models Before You Ship
• Prompt Management and Versioning in Production
• How to Choose the Right AI Model: A Practical Model-Routing Guide
• A/B Testing Prompts and Models in Production

A retry is a write, not a read. Resending after a timeout asserts that the first attempt either never executed or produced no effect you care about, and against LLM APIs that is usually false. Correct retries need two mechanisms: a classifier that separates provably-unsent failures from unknown-outcome ones, and an idempotency key that collapses duplicate attempts into one execution. Backoff, jitter and budgets only stop you making an outage worse while you wait.

Why a client-side timeout tells you nothing about the server

Your HTTP client gives up at 30 seconds. The provider does not know that. For a non-streaming completion the request is already sent, the model is mid-generation, and nothing propagates your disconnect into the inference worker. The generation finishes, the output tokens are counted, and you are billed for all of them. You received nothing. In your own data this shows up as billed completion tokens with no delivered response, and it means your retry policy is paying twice.

Worked example. A model at $3.00 per million output tokens, completions averaging 1,800 tokens, a client timeout at 30 s against a real server-side latency of 34 s. One affected request costs 1,800 / 1,000,000 x $3.00 = $0.0054 and returns zero bytes. Two retries cost 2 x $0.0054 = $0.0108. At 20,000 affected requests a day that is $216 daily, roughly $6,480 a month, for nothing.

A failure taxonomy you can actually code against

Retry decisions turn on four axes: transport outcome, HTTP status, provider error type, and semantics. Broken retry code inspects one, usually the status code.

Transport failures

DNS failure, connection refused and TLS handshake failure are the only provably pre-send failures: no HTTP request was written, so nothing can have executed. Everything else is unknown-outcome. A reset before response headers is unknown: an edge proxy can accept and forward before resetting. The one question that matters: did any response byte arrive?

HTTP status classes

408, 429, 500, 502, 503 and 504 are retryable. 400, 401, 403, 404, 413 and 422 are not, because identical bytes produce an identical rejection. 401 gets one exception: a single retry after a credential refresh that changed the credential. Never treat any 5xx as retryable without reading the body.

Provider-specific overload signals

This is where generic libraries fail. OpenAI returns 429 for both rate limiting and quota exhaustion, and the two need opposite responses: rate_limit_exceeded is retryable, insufficient_quota is a billing wall that never clears. Same status, opposite decision, distinguishable only from the body’s type field. Anthropic uses 529 overloaded_error, outside the standard set, so a status in {500, 502, 503, 504} check misses it. Vertex and Gemini return RESOURCE_EXHAUSTED and UNAVAILABLE in a gRPC envelope, and some streaming paths deliver the error in-band on HTTP 200. Bedrock raises ThrottlingException.

That compounds badly: the OpenAI Python SDK retries twice by default, Anthropic’s twice, boto3’s adaptive mode more. Add an application layer with three attempts and one logical operation produces 3 x 4 = 12 upstream calls. Two retry layers multiply; they do not add.

Semantic failures

Content-filter blocks, context-length overflows, refusals and schema violations are deterministic in the input, so retrying is pure cost. At temperature above zero a filter false-positive is stochastic, and re-issuing at temperature zero can pass. That is a new request with different parameters, not a retry.

Condition	Retryable?	Action
DNS failure, connection refused, TLS handshake failure	Yes, safely	Jittered retry; provably pre-send, no duplicate risk
Connection reset before response headers	Yes, unknown outcome	Retry only with an idempotency key
Read timeout after headers received	Unknown	Treat as executed; require key, log orphan tokens
408 Request Timeout	Unknown	Key required; honour any Retry-After
429 rate_limit_exceeded	Yes	Honour Retry-After, jitter, reduce concurrency
429 insufficient_quota	No	Fail fast, page the billing owner
500 Internal Server Error	Usually	Jittered retry; open breaker if sustained
502 Bad Gateway	Yes	One immediate retry, then backoff
503 / 504	Yes, unknown outcome	Backoff plus key; shed load if persistent
529 overloaded_error (Anthropic)	Yes	Long backoff, expect minutes not seconds
ThrottlingException (Bedrock)	Yes	Disable SDK retries first, then own it
400 / 422 malformed request	No	Fix the caller; retrying wastes budget
401 / 403	No, unless refreshed	Refresh credential once, then fail
404 model not found	No	Fix model identifier or routing table
413 payload too large	No	Truncate, chunk, or switch model
Content filter / refusal	No	Different prompt or parameter set, new request
context_length_exceeded	No	Truncate or route to a longer-context model

Exponential backoff with full jitter

Ship full jitter: delay = uniform(0, min(cap, base * 2 ** attempt)), attempt zero-indexed. Worked example, base 0.5 s, cap 20 s. Attempts 0 to 5 cap the delay at 0.5, 1, 2, 4, 8 and 16 s, with means of 0.25, 0.5, 1, 2, 4 and 8 s. Attempt 6 onward caps at min(20, 64) = 20 s, mean 10 s. A three-attempt policy adds an expected 0.25 + 0.5 = 0.75 s of sleep.

Jitter exists because failures are correlated. Without it, every client that hit the same blip retries at the same instant and the retry wave recreates the overload. Deterministic backoff also synchronises with recovery: if a provider sheds load for two seconds, every client that picked a two-second delay returns exactly as the queue drains. AWS’s published simulation of these variants found full jitter won on both request count and completion time.

Set the cap against your request budget: a 20-second sleep inside a 30-second client timeout means the retry never executes. Cap interactive calls at 20 s and batch at 60 s, and enforce a wall-clock budget per logical operation including every sleep. Honour Retry-After when present, clamped to the cap. Retries also need a budget: a token bucket of 100 tokens refilling 0.1 per successful request bounds sustained retry traffic at roughly 10% of your success rate.

import random

RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}

# Overload signals that are retryable despite non-standard codes or envelopes.
RETRYABLE_TYPES = {
    "rate_limit_exceeded", "overloaded_error", "server_error",
    "ThrottlingException", "RESOURCE_EXHAUSTED", "UNAVAILABLE",
}

# Signals that share a status code with a retryable error but never clear.
FATAL_TYPES = {
    "insufficient_quota", "billing_hard_limit_reached", "invalid_api_key",
    "invalid_request_error", "content_policy_violation",
    "context_length_exceeded",
}

class RetryBudget:
    """Token bucket that caps retries at a fraction of successful traffic."""

    def __init__(self, capacity: int = 100, refill_per_success: float = 0.1):
        self.capacity = capacity
        self.tokens = float(capacity)
        self.refill = refill_per_success

    def grant(self) -> bool:
        if self.tokens < 1.0:
            return False
        self.tokens -= 1.0
        return True

    def on_success(self) -> None:
        self.tokens = min(self.capacity, self.tokens + self.refill)

def classify(status=None, provider_type=None, exc=None, headers_received=False):
    """Return (retryable, provably_pre_send, reason)."""
    if exc is not None:
        if isinstance(exc, TimeoutError):
            # No headers means the request may or may not have executed.
            return (not headers_received), False, "timeout"
        if isinstance(exc, ConnectionRefusedError) and not headers_received:
            return True, True, "connection_refused"
        if isinstance(exc, OSError) and not headers_received:
            return True, False, "transport_error"
        return False, False, "unexpected_exception"

    if provider_type in FATAL_TYPES:
        return False, False, "provider:" + str(provider_type)
    if status in (401, 403):
        return False, False, "auth"
    if status is not None and 400 <= status < 500 and status not in RETRYABLE_STATUS:
        return False, False, "client_error_" + str(status)
    if status in RETRYABLE_STATUS or provider_type in RETRYABLE_TYPES:
        return True, False, "status_" + str(status)
    return False, False, "unclassified_" + str(status)

def full_jitter(attempt: int, base: float = 0.5, cap: float = 20.0) -> float:
    """attempt is zero-indexed: uniform(0, min(cap, base * 2**attempt))."""
    return random.uniform(0.0, min(cap, base * (2 ** attempt)))

Idempotency keys: one key per logical operation

The key identifies intent, not an attempt. Generate it once before the first send, persist it, and reuse it byte-for-byte on every retry. Where the operation has a business identity, derive it deterministically: sha256(tenant_id + "generate_summary" + document_id + revision_id) survives restarts, deployments and queue redelivery. Where no identity exists, generate a UUIDv7 or ULID once and store it in the row that represents the operation.

Do not hash the request body: two identical prompts from different tenants are different operations, and a retry that re-serialises the message list in a different order would compute a different key. Store a body fingerprint beside the key so reuse with a changed payload is a hard error. Keep state in Redis with SET key state NX EX 86400: in_progress with a short lease so a crashed worker cannot wedge the key, completed with the response body for transparent replays, failed with the terminal error class. The TTL must exceed the worst-case client retry window.

Replay semantics must be explicit. Completed: return the stored response with its original status plus a marker header such as Idempotency-Replayed: true. In flight: 409 Conflict or 202 with a polling location, never blocking the second request on the first. Same key, different fingerprint: 422, hard fail. Terminally failed: store the error class but not the response, and let the caller re-key after fixing the input.

Provider support is uneven: some providers document an idempotency header on some endpoints, many document nothing, and support differs between streaming and non-streaming paths. Terminate idempotency in your own layer, so one component owns the dedupe table and forwards at most one upstream dispatch per key. That is the strongest argument for putting a gateway in front of providers, whether qoraapi.com or a proxy you run yourself: it becomes the single place the key is enforced. A gateway guarantees at-most-once dispatch, not at-most-once execution; if it times out upstream, the outcome is still unknown.

import { createHash, randomUUID } from "node:crypto";

type Stored = {
  state: "in_progress" | "completed" | "failed";
  fingerprint: string;
  status?: number;
  body?: unknown;
  errorClass?: string;
};

const TTL_SECONDS = 86_400;

export function deriveKey(tenantId: string, operation: string, eventId: string): string {
  // Stable across restarts, deployments and queue redelivery.
  return createHash("sha256")
    .update(`${tenantId}:${operation}:${eventId}`)
    .digest("hex");
}

export function newKey(): string {
  // No natural business identity: generate once, persist with the operation row.
  return randomUUID();
}

function fingerprint(payload: unknown): string {
  return createHash("sha256").update(JSON.stringify(payload)).digest("hex");
}

export async function withIdempotency(
  redis: any,
  key: string,
  payload: unknown,
  exec: () => Promise<{ status: number; body: unknown }>,
) {
  const fp = fingerprint(payload);
  const claim = await redis.set(
    `idem:${key}`,
    JSON.stringify({ state: "in_progress", fingerprint: fp } satisfies Stored),
    "NX",
    "EX",
    TTL_SECONDS,
  );

  if (claim === null) {
    const prev: Stored = JSON.parse(await redis.get(`idem:${key}`));

    if (prev.fingerprint !== fp) {
      return { status: 422, body: { error: "idempotency_key_reused_with_different_payload" } };
    }
    if (prev.state === "in_progress") {
      return { status: 409, body: { error: "operation_in_progress", key } };
    }
    if (prev.state === "completed") {
      return { status: prev.status ?? 200, body: prev.body, replayed: true };
    }
    return { status: 409, body: { error: "previous_attempt_failed", errorClass: prev.errorClass } };
  }

  try {
    const result = await exec();
    await redis.set(
      `idem:${key}`,
      JSON.stringify({ state: "completed", fingerprint: fp, ...result } satisfies Stored),
      "EX",
      TTL_SECONDS,
    );
    return result;
  } catch (err: any) {
    await redis.set(
      `idem:${key}`,
      JSON.stringify({
        state: "failed",
        fingerprint: fp,
        errorClass: err?.code ?? "unknown",
      } satisfies Stored),
      "EX",
      TTL_SECONDS,
    );
    throw err;
  }
}

Idempotency for streaming responses

A partially consumed stream cannot be retried transparently, for three independent reasons. Sampling is stochastic, so a retry produces a different completion and you cannot dedupe by comparing text. Token accounting arrives in the terminal chunk, so an aborted stream leaves you with no authoritative billing number. And whether the provider keeps generating after your disconnect varies by provider.

Resume works only when the provider exposes a stable response id plus a continuation endpoint, which most do not. Last-Event-ID resumption is a property of your own SSE stream, not the provider’s token stream: you can resume delivery from a server-side buffer, but only if you kept the buffer. Restart with client-side dedupe suits machine consumers that can discard a partial result, and is wrong for chat UIs where the user already watched tokens appear. Accepting the loss means marking the message failed, regenerating, and recording the duplicate spend. What I would ship: buffer server-side, do not emit until the first chunk is committed, persist accumulated text against the key, and on retry offer an explicit regenerate action rather than a silent retry. Delivery mechanics are in AI API streaming and SSE.

Duplicate side effects are worse than duplicate spend

Money is recoverable. A sent email, a created ticket or a shipped order is not. Every tool that writes, sends or charges is non-idempotent by default, and a model that sees a tool error will call it again.

Use natural keys where a business identity exists: sha256(tenant_id + "send_invoice_email" + invoice_id) is stable across restarts, deployments and replays. Otherwise the dedupe table is the primitive that works. A unique constraint plus INSERT ... ON CONFLICT DO NOTHING with a rowcount check gives exactly one winner under concurrency; an in-process lock or a Redis GET-then-SET has a race window and fails precisely when two workers retry simultaneously.

For effects that must be atomic with a database state change, use the outbox pattern: write the intent into the outbox in the same transaction as the state change, drain it with at-least-once delivery, and make the downstream effect idempotent by passing your key onward. For irreversible external effects, use two-phase confirmation: insert a pending row carrying your key, call the downstream API with that key, then mark committed with the downstream identifier. A timeout leaves a pending row that tells you what to reconcile. Two rules for review: the key never comes from model output, and retry permission is declared by the tool author rather than inferred. Related patterns are in AI agents and tool use.

Circuit breakers, bulkheads and hedging

If every request is failing and each retries three times, you have tripled load on a system already shedding. A provider’s 429 is an instruction to reduce concurrency, not increase attempts. Aggregate retries without concurrency control are a self-inflicted denial of service.

Open the breaker on either N consecutive failures or a failure ratio above a threshold over a rolling window, with a minimum sample of 20 requests. Stay open 30 seconds initially, longer for provider-wide outages. Half-open admits exactly one probe, and the probe should be cheap, a models listing or a one-token completion rather than a real user request. If it fails, multiply the cooldown (30 s, 60 s, 120 s, capped at 5 minutes) instead of re-probing every 30 seconds.

Bulkheads are the other half: a semaphore per provider limiting in-flight requests, which retries must acquire through. If the retry path bypasses the cap, retries become the load. On 429s, halve the limit and recover at roughly 10% per minute. Use separate connection pools per provider so one provider’s slow responses cannot starve another’s sockets. The multi-provider version is in AI API failover across multiple providers.

Hedging sends a duplicate to a second provider after a latency threshold, takes the first response, and cancels the loser. It works when the tail is dominated by queueing rather than work. Set the threshold near your p95: if p50 is 1.2 s and p95 is 3.5 s, hedge at 3.5 s, so only a few percent of requests spawn a second call. It is wrong for anything with side effects, because hedging is a duplicate by design, and wrong when the slow request is slow because the model is reasoning: cancelling at 3.5 s discards a 40-second generation and you pay for the tokens already produced. Cancellation is not a refund.

Cost arithmetic. 1,000,000 daily requests, a 5% hedge rate, so 50,000 hedges. If 60% return first, you still pay for the original’s partial generation. Worst case you pay for 1,050,000 generations instead of 1,000,000: a 5% spend increase for a tail-latency win. Good trade for a user-facing chat; not for a batch job where nobody is waiting.

A retry helper that classifies, budgets, and refuses unkeyed retries

import asyncio
import time
from dataclasses import dataclass

class RetryBudgetExhausted(RuntimeError):
    pass

@dataclass
class AttemptLog:
    logical_id: str
    attempt: int
    reason: str
    status: int | None
    provider_type: str | None
    retry_after: float | None
    delay_ms: int
    outcome: str

def extract_error_fields(exc):
    """Pull status, provider error type and Retry-After out of an SDK exception."""
    status = getattr(exc, "status_code", None) or getattr(exc, "http_status", None)
    ptype = None
    body = getattr(exc, "body", None)
    if isinstance(body, dict):
        err = body.get("error", body)
        if isinstance(err, dict):
            ptype = err.get("type") or err.get("code")
    if ptype is None:
        ptype = getattr(exc, "code", None)

    retry_after = None
    headers = getattr(exc, "headers", None) or getattr(exc, "response", None)
    if hasattr(headers, "get"):
        raw = headers.get("retry-after")
        if raw is not None:
            try:
                retry_after = float(raw)
            except (TypeError, ValueError):
                retry_after = None
    return status, ptype, retry_after

async def call_with_retries(
    fn,
    *,
    logical_id: str,
    idempotency_key: str | None,
    budget: RetryBudget,
    max_attempts: int = 3,
    base: float = 0.5,
    cap: float = 20.0,
    wall_clock_budget: float = 25.0,
    allow_unkeyed_pre_send_retry: bool = True,
    log=None,
):
    """fn is called as fn(idempotency_key). An unkeyed write is never retried
    unless the failure is provably pre-send."""
    if idempotency_key is None and not allow_unkeyed_pre_send_retry:
        max_attempts = 1

    log = log or (lambda a: None)
    deadline = time.monotonic() + wall_clock_budget
    last_exc = None

    for attempt in range(max_attempts):
        try:
            result = await fn(idempotency_key)
            budget.on_success()
            return result
        except Exception as exc:  # noqa: BLE001 - classification is explicit
            last_exc = exc
            status, ptype, retry_after = extract_error_fields(exc)
            headers_received = status is not None
            retryable, pre_send, reason = classify(status, ptype, exc, headers_received)

            if not retryable:
                raise
            if attempt == max_attempts - 1:
                raise
            if idempotency_key is None and not pre_send:
                # Unknown outcome with no key: retrying risks a duplicate effect.
                raise
            if not budget.grant():
                raise RetryBudgetExhausted(reason) from exc

            delay = retry_after if retry_after is not None else full_jitter(attempt, base, cap)
            delay = min(delay, cap)
            if time.monotonic() + delay > deadline:
                raise

            log(
                AttemptLog(
                    logical_id=logical_id,
                    attempt=attempt,
                    reason=reason,
                    status=status,
                    provider_type=ptype,
                    retry_after=retry_after,
                    delay_ms=int(delay * 1000),
                    outcome="retry",
                )
            )
            await asyncio.sleep(delay)

    raise last_exc

Three properties make this correct. Classification happens before any retry decision, so a 429 carrying insufficient_quota never consumes budget and never adds load. The budget is checked before sleeping, so amplification is bounded even when everything is failing. And the unkeyed case is gated on pre_send, so a non-idempotent call retries only when the classifier can prove no bytes reached the provider. Pair it with max_retries=0 on the SDK so exactly one retry layer exists.

What to log per attempt

Log one record per attempt, not per logical request. The fields that earn their keep: logical request id, attempt number, idempotency key, provider, model, endpoint, error class, HTTP status, provider error type, the provider’s own request id from the x-request-id or request-id header, any Retry-After, computed delay, prompt and completion tokens, cost, duration, and outcome. That provider request id is your only handle in a billing dispute.

From those records derive five metrics: retry ratio (attempts divided by logical operations), amplification (upstream calls divided by logical operations), retry success ratio, duplicate effects served from the dedupe table, and orphaned tokens, meaning billed tokens with no delivered response. These separate two situations that look identical in a success-rate dashboard. One flaky request: low retry ratio, amplification near 1.0x, high retry success ratio, normal latency. A masked outage: climbing retry ratio and amplification, falling retry success ratio, and rising p50 and p99 on requests that ultimately succeed, because backoff sleeps sit inside the request. Your error rate can look flat while the service is three times slower: that is the state worth alerting on. Metric design for this layer is covered in LLM observability.

The default retry policy I would ship

• Exactly one retry layer. Set SDK retries to zero and own the policy centrally, so amplification is your policy rather than the product of two.
• Retry on connection refused, DNS failure, TLS failure, 408, 429 rate-limit, 500, 502, 503, 504, and provider overload types (529 overloaded_error, ThrottlingException, RESOURCE_EXHAUSTED, UNAVAILABLE).
• Never retry 400, 403, 404, 413, 422, content-filter blocks, context-length overflows, or quota and billing errors. Allow one retry on 401, only after a credential refresh that changed the credential.
• Three attempts for interactive calls, five for batch. Full jitter, base 500 ms, cap 20 s interactive and 60 s batch, with a 25 s wall-clock budget per logical operation including every sleep.
• Honour Retry-After whenever present, clamped to the cap.
• An idempotency key is mandatory for every request that can bill or cause a side effect. Without a key, at most one retry, and only for provably pre-send failures.
• A retry budget token bucket of 100 tokens refilling at 0.1 per successful logical request, checked before every sleep.
• One breaker per provider: 20-request minimum sample, open at 50% failures, 30 s cooldown with exponential backoff capped at 5 minutes, one cheap half-open probe.
• One in-flight semaphore per provider, which retries must acquire through. Halve the limit on 429s, recover at roughly 10% per minute.
• Never retry a partially consumed stream. Never hedge a non-idempotent call. Never let a model generate an idempotency key.

That policy is deliberately conservative about retrying and aggressive about classifying. Most teams I review have the ratio inverted: a generic retry decorator everywhere, no classification, no key. The result looks resilient in staging and doubles the inference bill during the first real incident.

Frequently asked questions

Is a 500 always safe to retry?

No. A 500 can arrive after the request was fully processed and the response failed on the way out, which makes it unknown-outcome rather than pre-send. Some gateways also return 500 wrapping an upstream 400 or a content-filter rejection. Treat 500 as retryable only when the body’s error type indicates a server-side fault, and require a key.

Should the idempotency key be a UUID?

A UUID is fine for uniqueness but not sufficient. The key must be generated once per logical operation and persisted before the first send. A UUIDv4 generated inside the function that makes the HTTP call is regenerated on every retry, which makes it useless. Prefer UUIDv7 or ULID for sortability, and prefer derivation from a business event id.

Can I just retry the whole agent run instead of individual steps?

No. An agent run is a sequence of steps, some with side effects, so retrying the run re-executes every completed step and duplicates each effect. Retry at the step level with a key derived from the run id and step index, and treat the run as a state machine that resumes from the last committed step. A step that cannot be made idempotent needs a compensating action, not a retry.

Does my provider deduplicate retries automatically?

Assume not. Idempotency header support varies by provider, by endpoint, and often between streaming and non-streaming paths. Even where a header exists, verify it experimentally, because a silently ignored header looks exactly like a working one until you inspect the bill.

How long should the idempotency record live?

Longer than the worst-case window in which a client could replay: your maximum client timeout multiplied by your maximum attempts, plus queue delay, plus clock skew. A 30 s timeout with three attempts and 60 s of queueing gives roughly 150 s. Twenty-four hours of headroom is conventional because it also covers a restarted worker or a stuck batch job.

Conclusion

Retry correctness is a classification problem first and a timing problem second. Decide whether an attempt could have executed before deciding how long to wait, persist one idempotency key per logical operation before the first byte leaves, and cap the blast radius with a budget, a breaker and a per-provider semaphore. The backoff formula is the easy part; the classifier and the key are where duplicate generations, duplicate tool calls and double charges are actually prevented. If you would rather not maintain the dedupe table, the classification rules and the breaker state yourself, that is precisely the layer a gateway such as qoraapi.com can own, provided you understand that it bounds duplicate dispatch and not duplicate execution. Start with three attempts, full jitter, a mandatory key on every write, and an alert on amplification above 1.2.

Related reading

• How to Handle AI API Rate Limits and 429 Errors
• How to Build a Multi-Provider AI Failover Layer for 99.9% Uptime
• Async AI APIs: Job Queues, Webhooks, and Long-Running Tasks
• Building Reliable AI Agents: Guardrails, Retries, and Human-in-the-Loop

A web-grounded answer is not a smarter model call; it is a retrieval pipeline with a model at both ends. Your code decides whether a search is warranted, generates and executes queries, fetches and filters evidence, and only then lets the model write prose over a small, dated evidence set, with citations attached in code rather than by the model. When “the LLM gave me outdated information”, the failure is almost always in one of those deterministic stages.

Parametric knowledge is the wrong tool for time-sensitive questions

Weights encode text observed up to a training cutoff, which makes them excellent at stable knowledge — language semantics, algorithm design, protocol structure — and incapable of holding a fact whose truth value changes faster than the model retrains. A model cannot know a value is stale, because staleness is not a property of the text it learned from.

Three failure shapes follow, each needing a different guard.

• Confidently stale. The weights hold a value that was correct at training time and is now wrong, stated with the certainty the model applies to arithmetic. Nothing marks it as having moved.
• Confidently wrong. The model interpolates a plausible value that was never true: a version number that never existed, a parameter name that sounds right. Fabrication with correct surface form.
• Silently missing. The entity or event postdates the cutoff, so there is nothing in memory. The model refuses, hallucinates, or answers a nearby question.

Staleness is a freshness problem: fix it with dated retrieval and a window. Wrongness is a verification problem: fix it with entailment checks against retrieved evidence. Missingness is a routing problem: force retrieval for questions needing post-cutoff knowledge. This is not private-corpus RAG — chunking and vector-store choices for your own documents are covered in Production RAG architecture. Here the corpus is the open web: uncontrolled, and full of sloppy content.

The grounding loop, step by step

Eight steps, in order. The interesting engineering is deciding which are model-driven and which must be deterministic, because anything you leave to the model is something you cannot test.

Step	Driven by	Deterministic obligation
1. Route: is a search needed?	Heuristic plus small classifier	Bypass retrieval for stable-knowledge and text-transformation questions
2. Generate queries	Small model, structured output	Cap the count, reject empty or duplicate queries, inject the current date
3. Execute searches	Code	Timeouts, per-request budget, dedupe by URL, log raw results
4. Fetch and extract	Code	Fetch only top-N candidates; strip boilerplate; keep a content hash for caching
5. Select evidence	Hard filters in code, ranking model optional	Recency window, domain allowlist, one document per domain, hard drop of the rest
6. Compose the answer	Frontier model	Evidence-only prompt, span ids instead of URLs, temperature 0
7. Attach citations	Code	Map span ids to URLs from your evidence list; never render a model-authored link
8. Verify	Code, optionally a small entailment model	Every cited sentence must be supported by the cited span, or it is dropped or flagged

The model proposes, the code disposes. Evidence selection is a decision; citations are a rendering step.

Query generation: a raw question is a bad search query

Users write questions for a human who shares their context; search engines need standalone, keyword-bearing queries. Three things break in translation. Conversational deixis (“my plan”, “this error”) is unresolvable by an index. Question phrasing (“is it still required?”) rarely matches document phrasing (“obligation applies from”). And most non-trivial questions are multi-hop, needing two or three lookups, each with its own query.

Take a real question: Is the EU AI Act’s general-purpose AI obligation already in force for models we shipped last year? That is three lookups — the obligation’s start date, the transition rule for models already on the market, and whether “shipped last year” falls inside it.

• EU AI Act general purpose AI obligations application date 2026
• EU AI Act GPAI transition period models placed on market before August 2025
• site:eur-lex.europa.eu AI Act Article 113 entry into force

Two to four queries is the right band. Beyond that you are not improving recall; you are flooding the evidence set with near-duplicates and third-party restatements, pushing the primary source out of the top-N you can afford to fetch. Over-searching also manufactures conflicts you must then resolve.

import json
from dataclasses import dataclass, field

QUERY_SYSTEM = """You convert a user question into web search queries.
Rules:
- Emit 2 to 4 queries. Never emit a single query.
- Each query must stand alone: no pronouns, no reference to earlier turns.
- Include at least one query targeting a primary source (regulator, vendor, standards body).
- If the question is time dependent, put the explicit year or window in the query.
- If no web lookup is needed, set needs_search to false and queries to [].
Return JSON only, matching: {"needs_search": bool, "queries": [str], "freshness_days": int}
freshness_days is how old an acceptable source may be for this question."""

@dataclass
class QueryPlan:
    needs_search: bool
    queries: list
    freshness_days: int
    notes: list = field(default_factory=list)

def plan_queries(client, question, today_iso):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": QUERY_SYSTEM},
            {"role": "user", "content": "Today is %s. Question: %s" % (today_iso, question)},
        ],
    )
    raw = json.loads(resp.choices[0].message.content)

    seen, queries = set(), []
    for q in raw.get("queries", []):
        q = " ".join(str(q).split())
        if len(q) < 6 or q.lower() in seen:
            continue
        seen.add(q.lower())
        queries.append(q)
    queries = queries[:4]

    notes = []
    if raw.get("needs_search") and not queries:
        notes.append("planner requested search but produced no usable query")
    return QueryPlan(
        needs_search=bool(raw.get("needs_search")) and len(queries) > 0,
        queries=queries,
        freshness_days=max(1, min(int(raw.get("freshness_days") or 30), 3650)),
        notes=notes,
    )

The model writes the queries; the code caps, dedupes and sanitises them. A planner that claims it needs a search but returns nothing usable is a routing bug for your logs, not a silent fallback to an ungrounded call.

Search versus fetch: the two-stage pattern

A search API returns snippets — a title, a URL, a date if you are lucky, and 150-300 characters of text. They are cheap and fast, and the worst of both worlds for grounding: stale relative to the live page, truncated mid-sentence, and ranked for click-through, which rewards keyword-stuffed pages with absent or wrong dates. Fetching gives you real content at a price: 200-1500 ms per fetch, tens to hundreds of kilobytes of HTML, and 10-30% of pages unparseable — JavaScript shells, cookie walls, paywalls, infinite-scroll docs.

So run two stages. Execute the queries, collect every result, deduplicate by normalised URL, keep the best rank per URL. Then fetch only the top three to six candidates by your own ranking, not the search engine’s, and extract main content. Skip the fetch when the question is a single-hop lookup, the snippet contains the answer as a verifiable token — a version string, a date, a numeric limit — and the source is a primary domain.

Source quality: filter evidence before it reaches the model

Every document in the context window is a vote. Junk evidence does not dilute good evidence so much as hand the model a fluent, confident, wrong thing to summarise. Filtering belongs in code, because a language model judging source reliability from a 1,200-character excerpt is doing the task it is worst at.

Signal	How to compute it	How to use it
Recency	Structured data datePublished/dateModified, then meta tags, then a date in the first 400 characters	Hard drop outside a multiple of the question’s freshness window; unknown age is penalised, not trusted
Domain reputation	Allowlist of primary sources (regulator, vendor docs, standards body) plus an explicit denylist	Multiplicative boost; an allowlist match can satisfy a “primary source present” gate
States a date	Any parseable date, including a visible byline	A page that never says when it was written cannot establish freshness; cap its score
Cross-source corroboration	Same claim or same numeric value present on two or more independent domains	Required for high-stakes numeric claims; absent corroboration lowers confidence or triggers abstention
Extraction density	Extracted text length and boilerplate ratio after main-content extraction	Drop near-empty extractions; they are usually walls or shells
Source independence	Registrable domain of each result	Keep one document per domain, so “three sources” means three publishers rather than three syndications of one wire story

Rank, then drop. A pipeline that keeps twenty results and hopes the model sorts it out is not grounded; it is a summariser with extra steps. Three to six high-quality spans with provenance metadata attached produce better answers and cheaper ones.

Freshness: dates, undated pages, and misdiagnosed bugs

Freshness is something you compute, not something the model reports. Extract a publication or modification date in a fixed order of preference — structured data, then meta tags, then the HTTP Last-Modified header, then a date in the opening text — and record which source you used. A date from a copyright footer is not evidence of when the content changed, so treat it as weak.

Undated pages are the common case, not the exception. Treat them as unknown age rather than fresh, and score them middling: with a 30-day freshness window, an undated post about a pricing change is a rumour with a domain name, not a source. Where the requirement is hours rather than days — status pages, live inventory, release feeds — skip web search and hit the primary source’s own API, because the index lags by days.

This is where the most common misdiagnosis happens. When a user reports “the model answered with 2023 numbers”, the model almost certainly summarised its evidence faithfully. The bug is upstream: a generic query, a ranker that preferred an old high-authority page, a missing recency filter, a cached fetch. Debug the retrieval trace before touching the prompt; LLM observability covers the span structure that makes grounding failures legible.

Citations that actually work

Asking a model to produce citations produces citations that look right. Two failures hide behind that. The first is a fabricated URL: a plausible link the model never saw. The second is subtler and more common — the attribution gap: the URL is real, it was in the evidence, and the sentence it is attached to is not supported by that page. The citation resolves; it just does not support the claim.

The fix is structural: citations are attached in code, from your evidence list, never authored by the model.

1. Number the selected spans [S1], [S2], and pass them to the model with titles and dates.
2. Instruct the model to append the span ids it relied on after each factual sentence, and to write no URLs.
3. Post-process: if the model cites a nonexistent span id, discard the answer and retry with a tighter evidence set. Map surviving ids to URLs from your list.
4. Verify support: check each cited sentence is entailed by the cited span, using a lexical-overlap floor as a cheap first pass and a small entailment model for the rest.
5. Drop or mark sentences that fail verification instead of silently keeping them.

Step four is the one teams skip, and the one that turns “we show sources” into “our sources mean something”.

Conflict resolution

Sources disagree, and the disagreement is usually informative. Three responses are defensible: prefer the more authoritative and more recent source, and say so; surface the disagreement with both values and both dates; or abstain when the conflict is material to the user’s decision.

Silently picking one is the worst option, and it is the default of every pipeline that concatenates evidence and lets the model write. The model produces one number with full confidence, the user cannot tell whether that was consensus or a coin flip, and the error becomes undetectable. Detect conflict mechanically where you can: extract the numeric or named entity that answers the question from each span and compare. Where extraction is unreliable, ask a small model to classify the evidence as agreeing, conflicting or insufficient, and let code apply the policy.

When to abstain

Design the abstention path before the answer path, because the model will not choose it correctly on its own: asked to answer from evidence it almost always answers, and asked to refuse when unsure it refuses too often.

Make abstention a gate in code, evaluated before generation: at least two independent domains, at least one primary or allowlisted source, all evidence inside the freshness window, no unresolved material conflict. When the gate fails, return a structured non-answer — what was searched, what was found, which condition failed — plus the raw links so a human can finish the job. A confident wrong number is not a usable outcome, because the user cannot detect it. Track the abstention rate as a first-class metric: a step change is your earliest signal of a retrieval regression.

Cost and latency: worked arithmetic

Numbers below are a labelled worked example, not a benchmark. Assume $3.00 per million input tokens, $15.00 per million output tokens for the composing model, $0.005 per search API call, and no charge modelled for fetching.

Stage	Units	Cost
Ungrounded answer (1,200 in, 400 out)	1,200 x $3/M + 400 x $15/M	$0.0036 + $0.0060 = $0.0096
Query planning (600 in, 150 out, small model)	600 x $3/M + 150 x $15/M	$0.0018 + $0.0023 = $0.0041
Search calls	3 x $0.005	$0.0150
Fetched evidence if it reached the model (4 pages x ~2,500 tokens)	10,000 x $3/M	$0.0300, avoided by filtering in code
Composition (3,300 in, 500 out)	3,300 x $3/M + 500 x $15/M	$0.0099 + $0.0075 = $0.0174
Grounded total	$0.0041 + $0.0150 + $0.0174	$0.0365

So roughly 3.8x the cost of a plain call, with the search API alone contributing 41% of the grounded total. The table also shows the largest available saving: the 10,000 tokens of raw fetched content never need to enter a prompt. Filtering to three spans of about 800 tokens each removes $0.03 of token spend per answer, more than the search calls cost.

Latency is dominated by the network: 300-600 ms for planning, 200-800 ms for three parallel searches, 300-1,500 ms for four parallel fetches, and composition comparable to an ungrounded call. Add one to three seconds end to end, and stream.

Three caching layers pay for themselves. Cache search results keyed by normalised query with a TTL from the freshness class: minutes for status and pricing, a day or more for documentation. Cache fetched and extracted page text keyed by URL plus ETag or content hash — the biggest win, since it eliminates both the fetch and the re-extraction tokens. Cache final answers only when the freshness class tolerates it, because a semantically cached answer to a price question is stale by construction; see semantic caching for AI APIs. Route deliberately: a small model for planning and selection, the frontier model only for composition.

Evaluating groundedness separately from fluency

Fluency is free and it is not what you are shipping. A grounded answer can be fluent and wrong in a way no readability metric detects, so measure groundedness on its own labelled sample.

• Citation precision — of the citations emitted, what fraction support their sentence. Human-label a few hundred; this is the attribution gap measured directly.
• Citation coverage — what fraction of factual sentences carry a citation at all. Low coverage means the model is writing from memory inside a grounded prompt.
• Evidence recall — did the pipeline retrieve the authoritative source at all? Label the correct URL per question and check it appeared in the selected spans. This separates retrieval failure from composition failure.
• Freshness correctness — did the answer use the current value, from a source inside the required window?
• Abstention correctness — abstained when evidence was insufficient, answered when it was sufficient. Track both directions.

The labelled set has to be time-varying or it stops testing grounding. Pick questions whose answers genuinely move: a library’s latest stable release, a regulator’s filing deadline, a vendor’s entry-level price. Record the gold answer, the verification date and the gold source URL, then re-verify on a schedule. A static set decays into a memorisation test within months: the model answers from weights, your pipeline looks perfect, and the next question fails. Wire these metrics into the harness described in building an LLM eval harness.

A reference implementation

The pipeline below is the whole loop in miniature: plan, search, fetch-and-extract (stubbed), select with recency and domain filters, compose against numbered spans, then attach and verify citations in code.

import re
from datetime import datetime, timezone
from urllib.parse import urlparse

PRIMARY_DOMAINS = {"europa.eu", "eur-lex.europa.eu", "nist.gov", "ietf.org",
                   "w3.org", "docs.python.org", "developer.mozilla.org"}
MAX_EVIDENCE = 5
MIN_TEXT_CHARS = 400

CITE = re.compile(r"\[S(\d+)\]")

def domain_of(url):
    host = urlparse(url).netloc.lower()
    return host[4:] if host.startswith("www.") else host

def domain_trust(url):
    host = domain_of(url)
    for d in PRIMARY_DOMAINS:
        if host == d or host.endswith("." + d):
            return 1.0
    return 0.5

def parse_date(page):
    for key in ("dateModified", "datePublished"):
        if page.get(key):
            try:
                return datetime.fromisoformat(str(page[key]).replace("Z", "+00:00"))
            except ValueError:
                pass
    m = re.search(r"\b(20\d{2})-(\d{2})-(\d{2})\b", page.get("text", "")[:400])
    if m:
        return datetime(int(m[1]), int(m[2]), int(m[3]), tzinfo=timezone.utc)
    return None

def score(page, now, freshness_days):
    published = parse_date(page)
    if published is None:
        age_days, recency = None, 0.35      # unknown age is unverifiable, not fresh
    else:
        age_days = (now - published).days
        recency = max(0.0, 1.0 - age_days / float(max(freshness_days, 1)))
    density = min(1.0, len(page.get("text", "")) / 2000.0)
    return 0.55 * recency + 0.35 * domain_trust(page["url"]) + 0.10 * density, age_days

def select_evidence(pages, now, freshness_days):
    ranked = []
    for p in pages:
        if len(p.get("text", "")) < MIN_TEXT_CHARS:
            continue                        # boilerplate, a paywall, or a JS shell
        s, age = score(p, now, freshness_days)
        if age is not None and age > freshness_days * 3:
            continue                        # hard drop, not a soft penalty
        ranked.append((s, p))
    ranked.sort(key=lambda t: t[0], reverse=True)

    kept, seen_hosts = [], set()
    for s, p in ranked:
        host = domain_of(p["url"])
        if host in seen_hosts:
            continue                        # one document per domain keeps sources independent
        seen_hosts.add(host)
        kept.append({**p, "score": round(s, 3)})
        if len(kept) == MAX_EVIDENCE:
            break
    return kept

COMPOSE_SYSTEM = """Answer using ONLY the numbered evidence spans.
After every factual sentence, append the ids it relies on, e.g. [S1] or [S2][S3].
Write no URLs. Cite no span you did not use.
If the evidence does not support an answer, reply with exactly: INSUFFICIENT."""

def compose(client, question, evidence):
    blocks = [
        "[S%d] %s | published=%s\n%s" % (
            i, e["title"], e.get("published") or "unknown", e["text"][:1200])
        for i, e in enumerate(evidence, 1)
    ]
    resp = client.chat.completions.create(
        model="gpt-4o",
        temperature=0,
        messages=[
            {"role": "system", "content": COMPOSE_SYSTEM},
            {"role": "user", "content": "Evidence:\n\n%s\n\nQuestion: %s"
             % ("\n\n".join(blocks), question)},
        ],
    )
    return resp.choices[0].message.content.strip()

def attach_citations(answer, evidence):
    used = {int(i) for i in CITE.findall(answer)}
    if any(i < 1 or i > len(evidence) for i in used):
        return None, "cited a span that does not exist"
    sources = [
        {"id": i, "url": e["url"], "title": e["title"], "published": e.get("published")}
        for i, e in enumerate(evidence, 1) if i in used
    ]
    return {"answer": answer, "sources": sources}, None   # renderer links [S1] from this list

def unsupported_sentences(answer, evidence, min_overlap=0.12):
    problems = []
    for sentence in re.split(r"(?<=[.!?])\s+", answer):
        ids = [int(i) for i in CITE.findall(sentence)]
        if not ids:
            continue
        claim = set(re.findall(r"[a-z0-9]+", sentence.lower()))
        best = 0.0
        for i in ids:
            span = set(re.findall(r"[a-z0-9]+", evidence[i - 1]["text"].lower()))
            best = max(best, len(claim & span) / float(max(len(claim), 1)))
        if best < min_overlap:
            problems.append(sentence)
    return problems

def answer_grounded(client, search_fn, fetch_fn, question, now=None):
    now = now or datetime.now(timezone.utc)
    plan = plan_queries(client, question, now.date().isoformat())
    if not plan.needs_search:
        return {"mode": "ungrounded", "answer": plain_answer(client, question)}

    pages = []
    for q in plan.queries:
        for hit in search_fn(q, limit=5):
            pages.append({**hit, "text": fetch_fn(hit["url"])})

    evidence = select_evidence(pages, now, plan.freshness_days)
    if len({domain_of(e["url"]) for e in evidence}) < 2:
        return {"mode": "abstained", "reason": "fewer than two independent sources",
                "searched": plan.queries}

    draft = compose(client, question, evidence)
    if draft == "INSUFFICIENT":
        return {"mode": "abstained", "reason": "evidence did not support an answer",
                "searched": plan.queries}

    result, err = attach_citations(draft, evidence)
    if err:
        return {"mode": "abstained", "reason": err, "searched": plan.queries}
    result["unverified"] = unsupported_sentences(draft, evidence)
    result["mode"] = "grounded"
    return result

When web grounding is worth the complexity

Ground it when the truth value of the answer has a shorter shelf life than your deployment cadence: pricing, release versions, availability, regulatory status, deadlines, limits. Ground it when a citation is part of the product requirement — support replies that must be auditable, compliance-adjacent answers, anything a customer forwards onward.

Do not ground it when the question is about stable concepts: language semantics, algorithm design, historical facts. Those are what the weights are for, and a search round trip adds seconds and failure modes for nothing. Do not ground when the latency budget is under a second, or when the task is generative rather than factual — summarising a provided document, writing code against a spec. And do not ground against the open web what you already own: if your product answers questions about your own API, index your own docs and retrieve over them.

The pragmatic default is a router, not a policy. Classify the question, ground the time-sensitive and citation-requiring slice, send everything else straight through. Built against a single gateway key rather than each provider separately, qoraapi.com exposes one OpenAI-compatible endpoint with per-request cost visibility, which makes the arithmetic above verifiable in production rather than theoretical.

Frequently asked questions

Does giving the model a search tool make its answers grounded?

No. Tool use gives the model a lever, not a pipeline. A model with a search tool still chooses its own queries, reads unfiltered snippets, judges source credibility itself and writes its own citations — the decisions you cannot test. Grounding comes from the deterministic stages: recency and domain filtering, one-document-per-domain selection, citations rendered from your evidence list, a verification pass that drops unsupported sentences.

How many search queries should I generate per question?

Two to four for most questions. One fails on multi-hop questions and on questions whose phrasing does not match document phrasing. Beyond four you are mostly retrieving near-duplicates of the same claim, crowding the top-N candidates you can afford to fetch. Measure evidence recall — the fraction of questions where the labelled authoritative URL appears in your spans — and add queries only where it is low.

Why not just use a newer model with a later cutoff?

Because a cutoff moves; it does not disappear. A model trained through next quarter is still wrong about a release published next week, and the failure has the same shape. Freshness is a structural property of a static artefact answering a dynamic question. Newer models help with the confidently-wrong case, since better calibration means more refusals instead of fabrications, but the confidently-stale case is unaffected.

What should I do with pages that have no publication date?

Treat unknown age as a distinct freshness class rather than assuming fresh or discarding outright. Score it below a dated source from the same domain, admit it only when the question’s freshness requirement is loose, and never let it satisfy a “recent source present” gate alone. Where freshness is the whole point, an undated page is an assertion you cannot place in time.

Can the model just tell me which sources it used?

It will, and the answer will look plausible. A model-generated citation is a generated string, so it can resolve to a real page that does not support the sentence, which is the attribution gap, or to a page you never retrieved. Attach citations in code from evidence you actually fetched, force the model to reference span ids instead of URLs, and verify support after generation.

Conclusion

Web grounding is a retrieval engineering problem wearing a model costume. The model does two things well: turning a question into queries, and writing prose over a small set of curated spans. Everything between those points — routing, fetching, extraction, recency and domain filtering, evidence selection, citation attachment, support verification, the abstention gate — should be code you can read, test and trace.

The reason to build it is not accuracy in general; it is that time-sensitive facts are not knowable from weights, and a confidently wrong answer is worse than a visible gap. The reason not to build it is that it costs roughly four times an ungrounded call, adds one to three seconds, and introduces a failure class that only appears when a source changes under you. Decide per question class, not per product.

Related reading

• Building Production RAG: Chunking, Hybrid Search, and Re-Ranking
• Building a RAG Ingestion Pipeline: Crawling, Parsing, and Syncing
• Detecting and Reducing Hallucinations in Production LLM Apps
• AI Embeddings Explained: Vectors, Similarity, and Building Your First RAG

Synchronous request/response stops working the moment a model call can outlive the infrastructure in front of it. A 60-second load-balancer idle timeout, a 29-second serverless integration limit, or a mobile client that switches networks mid-request will kill a generation that is running fine on the provider side. Those limits are not yours to raise, so decouple submission from completion: the client gets a durable job id immediately, and the result arrives later by poll or webhook.

Why synchronous request/response breaks down

Every layer between your user and the model has its own clock, and the smallest one wins. nginx defaults proxy_read_timeout to 60 seconds. An AWS ALB idles connections out at 60 seconds. API Gateway enforces a hard 29-second integration timeout you cannot raise. None of them knows the generation is healthy; they terminate on wall-clock time. Serverless sharpens the mismatch: a Lambda can run for 15 minutes, but the front door in front of it caps at 29 seconds.

Mobile and unreliable clients

A phone that locks its screen, changes networks, or gets backgrounded drops the socket. The work is gone from the client’s perspective even though you already paid for it. Resuming requires a server-side identity that a synchronous endpoint cannot provide.

Generations that are legitimately slow

Summarising a 300-page contract, transcribing an hour of audio, generating video, or running an extended-reasoning model through a hard problem takes minutes. No prompt-engineering trick compresses a 90-second generation into a 30-second window. Batch pipelines hit this immediately: the larger the batch, the more certain some item blows the budget.

Three delivery patterns compared

There are exactly three shapes you can give an asynchronous-capable API, and each is correct under different constraints.

• Synchronous with streaming. One connection, tokens arriving as SSE chunks. Low time-to-first-token, unbounded duration, and a dropped connection loses the remainder unless the server supports resumption.
• Submit-and-poll. The POST returns 202 Accepted with a job id; the client polls GET /jobs/{id} until terminal. Trivial to implement and works from cron and a CLI.
• Submit-with-webhook. The server pushes the result to a URL you registered. Lowest latency, largest operational surface.

Pattern	Client complexity	Delivery latency	Failure behaviour	Works from	Best for
Synchronous + streaming	Low	Low to first token; unbounded to completion	Connection drop loses remaining output unless resumable	Any HTTP client, including browsers	Interactive chat, short completions
Submit-and-poll	Medium: retry and backoff logic in the client	Bounded by your poll interval	Client can always resume; state lives server-side	Any HTTP client, including CLI and cron	Batch work, internal tools, low volume, no public endpoint
Submit-with-webhook	High: public endpoint, HMAC verification, deduplication	Lowest: pushed the moment the job completes	Delivery depends on your endpoint being up; needs retries and a reconciliation sweep	Servers you operate	High-volume, user-facing long jobs

These are not mutually exclusive: a chat product streams, a document pipeline submits and webhooks, an internal report polls. Standardising on one shape is the mistake. An API gateway is the natural place to normalise the job contract so clients never learn which provider ran the work.

The job lifecycle state machine

Model the job explicitly rather than inferring it from a queue message. Six states are enough.

• queued — accepted and durable, not yet claimed.
• running — a specific worker holds a lease and is accountable for it.
• succeeded — terminal; the result is persisted and retrievable.
• failed — terminal; the error is recorded, no further attempts.
• cancelled — terminal; requested by the client.
• expired — terminal; outlived its deadline or its retention window.

Persist every transition

Write each transition to an append-only job_events table as well as updating current state on the job row. The row answers “where is it now” with one indexed read; the log answers “what actually happened” during an incident. Same split as any observability pipeline.

Why “running” must have a lease

A running state with no owner is a latent bug. If a worker crashes mid-call, a naive implementation leaves the job running forever: no retry, no alert, nothing for a reaper to match. Define running as “worker W holds a lease expiring at T”, renewed by a heartbeat, and a reaper reclaims any row where lease_expires_at < now().

Lease length is a real trade-off. Too short and a slow-but-healthy job is reclaimed and run twice; too long and a crashed worker blocks it for the full duration. A 60-second lease with a 15-second heartbeat tolerates three missed heartbeats, absorbing a GC pause without sluggish recovery.

Queue design

Choosing a broker

You probably do not need Kafka. For tens of thousands of jobs per day, Postgres with SELECT ... FOR UPDATE SKIP LOCKED is a complete transactional queue and removes a piece of infrastructure. It also lets the job row and the queue entry commit in the same transaction, which is the property you want.

Broker	Delivery semantics	Visibility timeout	Best fit
Postgres SKIP LOCKED	At-least-once	A column you manage yourself	Under 1M jobs/day; transactional enqueue, no extra infrastructure
Redis Streams	At-least-once	Pending entries list plus XCLAIM	High throughput, low latency, consumer groups
SQS	At-least-once; FIFO adds exactly-once processing within the queue	Native, extendable up to 12 hours	Managed, AWS-native, native dead-letter redrive
RabbitMQ	At-least-once	Native, per-consumer	Routing rules, priorities, per-message TTL

At-least-once is what you get

End-to-end exactly-once delivery does not exist. What you can build is exactly-once effects: at-least-once delivery plus idempotent consumers. SQS FIFO offers exactly-once processing within a queue via a five-minute dedup window, but that guarantee stops at the queue boundary.

Visibility timeout

A visibility timeout is the window during which a claimed message is hidden from other consumers. If it expires before you finish, the message is redelivered and a second worker starts the same job. It must exceed your p99 processing time, or the worker must extend it. SQS exposes ChangeMessageVisibility; on Postgres, the lease heartbeat.

Dead-letter queues

After N failed receives, move the message to a DLQ instead of retrying forever. A poison message that fails deterministically — malformed payload, retired model, empty account — otherwise consumes worker capacity indefinitely. Alarm on DLQ depth: an empty DLQ is healthy, one growing for an hour is an incident.

Never call the provider inside the transaction

This is the most common design error in job systems. The pattern looks reasonable: begin a transaction, insert the job row, call the model, update to succeeded, commit. It is wrong for three reasons.

1. It holds a transaction open for the entire external call. Connection pools are small, and a handful of slow generations exhausts them, stalling unrelated parts of the application.
2. If the call succeeds but the transaction rolls back, you have paid for a generation and thrown the result away.
3. If the transaction commits but the call fails, you have a job row asserting success with no result behind it.

Separate them. Commit the job row and an outbox event in one short transaction, then have a dispatcher publish after that commit. The outbox pattern makes persist-and-enqueue atomic, at the cost of a small bounded delay.

Webhook consumer correctness

A webhook endpoint is an unauthenticated public write path into your system. Treat it with the suspicion you would apply to any other one.

Verify the signature over the raw body

Compute the HMAC over the exact bytes you received, before JSON parsing or re-serialisation. Most verification bugs come from hashing a re-serialised object whose key order or whitespace no longer matches. Read the raw buffer, verify, then parse, and compare in constant time.

Reject replays with a timestamp window

Sign the timestamp with the payload and reject anything outside a tolerance window — five minutes is practical. Without a window, a captured request is valid forever, and the window must absorb clock skew and retry delay.

Deduplicate by event id

Every provider sends an event identifier. Insert it into a table with a unique constraint and treat a conflict as already processed. This is the only reliable defence against duplicate delivery: retries, network duplication and your own redeploys all produce them.

Return 2xx fast, then process

Do the minimum work in the handler: verify, deduplicate, enqueue, return 200. Everything else runs in a worker. If you process synchronously and exceed the provider’s delivery timeout — commonly 5 to 10 seconds — the provider marks delivery failed and retries, so you do the work twice while also being slow.

What happens when your endpoint is down

Providers retry with exponential backoff for hours to a few days, then give up — a few fast retries, widening intervals, then silence. Your endpoint must be idempotent, because it will see the same event at one second and again at six hours. Webhooks also cannot be your only source of truth.

Treat webhooks as the fast path and a periodic reconciliation sweep as the slow path. The sweep queries the provider for anything running long and settles it locally. The webhook makes the system fast; the sweep makes it correct.

Idempotency for long jobs

Long jobs are expensive, which makes duplicate execution expensive. Two layers of protection are needed.

Idempotency keys on submission

Let the client supply an idempotency key on the POST, stored under a unique constraint scoped to the tenant, so a retried submission returns the original job id instead of creating a second job. The POST will be retried whether you plan for it or not: by the client’s HTTP library, by a load balancer, or by a double-click.

Bind the key to a hash of the request body. If the same key arrives with a different body, return 409 Conflict rather than silently returning the first job’s result. Otherwise a client reusing a constant key gets plausible-looking answers to questions it never asked.

Provider idempotency support varies

Some providers accept an Idempotency-Key header on some endpoints. Others document none, or scope it to a short window or a subset of routes. You cannot depend on the provider to deduplicate your retries, so design as though every retry may produce a second generation and a second charge.

Make your own side effects safe

Every effect your worker performs needs a key that makes repeating it harmless.

• Result writes: INSERT ... ON CONFLICT (job_id) DO NOTHING.
• Billing: a ledger entry keyed by (job_id, 'completion') under a unique constraint, so a redelivered completion cannot double-charge. Same discipline as usage metering and billing.
• Notifications: deduplicate on (job_id, channel) before sending.
• Downstream calls: propagate the job id as the downstream idempotency key.

Where an effect cannot be made idempotent, record the provider’s request identifier as soon as you receive it and, on retry, query that job’s status instead of resubmitting. That converts a duplicate generation into a cheap status lookup.

Polling done right

Polling is not the inferior pattern. It is correct when you have no public endpoint, low volume, or a need for a single authoritative state store — it just has to be done with backoff.

Exponential backoff with jitter

Fixed-interval polling wastes requests early and is too slow late. Exponential backoff with full jitter — sleeping a uniform random amount between zero and min(cap, base * 2^attempt) — spreads load and cuts request volume by roughly an order of magnitude. Without jitter, clients that started together stay synchronised and reproduce identical thundering-herd bursts.

Honour Retry-After

When a status endpoint returns 429 or 503 with a Retry-After header, that value overrides your backoff. Ignoring it is how well-behaved clients get throttled into uselessness, and it causes the 429 storms that look like provider outages but are self-inflicted.

Long polling

Long polling holds the connection open until the result is ready or a server-side timeout of 30 to 60 seconds elapses, collapsing many status requests into one. Most LLM providers do not offer it, so in practice it applies at your own gateway.

The cost of polling at scale

Assume 50,000 jobs per day with an average completion time of 4 minutes.

Naive polling every 2 seconds: 240 divided by 2 gives 120 polls per job, so 6,000,000 status requests per day. Spread over 86,400 seconds that is about 69 requests per second — and since submissions cluster in business hours, the peak is several times that.

Apply backoff starting at 1 second, doubling, capped at 30 seconds. Cumulative poll times are 1, 3, 7, 15, 31, 61, 91, 121, 151, 181, 211 and 241 seconds — 12 polls to cover a 240-second job, roughly a 10x reduction. The same jobs now generate 600,000 requests per day, about 7 per second.

The architectural point is where those requests land. Clients should poll your job table, never the provider directly. Your status endpoint is one indexed primary-key read; a provider status call consumes rate-limit budget and may cost money. Keep one background reconciler as the only component that talks to the provider.

Progress reporting and cancellation

Progress reporting should be coarse and honest. Expose the state, an attempt count, and a percentage only if the provider actually reports one. A synthesised percentage that creeps toward 90 percent and then stalls is worse than a plain “running” label.

Cancellation support varies by provider and endpoint. Some accept a cancel call that terminates a running job. Some only cancel jobs that have not started. Some have none at all, and the generation completes and bills regardless. Establish which before putting a cancel button in a UI.

When a job cannot be cancelled, fall back to compensating actions: mark it cancelled locally so the result is discarded on arrival, stop downstream work, and refund if you billed at submission. Make sure the failover layer knows too, or a cancelled job gets failed over and billed twice.

A worker that claims a job with a lease

This worker uses Postgres as the queue. It claims one job atomically, records the lease owner, calls the model outside any transaction, and writes the result only if it still holds the lease — which stops a reclaimed job from being written twice.

import os
import time
import uuid
import psycopg
from openai import OpenAI

LEASE_SECONDS = 60
MAX_ATTEMPTS = 3

client = OpenAI(
    api_key=os.environ["QORA_API_KEY"],
    base_url="https://api.qoraapi.com/v1",
)

CLAIM_SQL = """
UPDATE jobs
   SET state = 'running',
       attempt = attempt + 1,
       lease_owner = %(worker)s,
       lease_expires_at = now() + make_interval(secs => %(lease)s),
       started_at = COALESCE(started_at, now())
 WHERE id = (
     SELECT id FROM jobs
      WHERE state = 'queued'
         OR (state = 'running' AND lease_expires_at < now())
      ORDER BY created_at
      FOR UPDATE SKIP LOCKED
      LIMIT 1)
RETURNING id, attempt, payload;
"""

FINISH_SQL = """
UPDATE jobs
   SET state = %(state)s,
       result = %(result)s,
       error = %(error)s,
       lease_owner = NULL,
       lease_expires_at = NULL,
       finished_at = now()
 WHERE id = %(id)s
   AND lease_owner = %(worker)s
RETURNING id;
"""

def claim(conn, worker):
    with conn.cursor() as cur:
        cur.execute(CLAIM_SQL, {"worker": worker, "lease": LEASE_SECONDS})
        return cur.fetchone()

def finish(conn, worker, job_id, state, result=None, error=None):
    with conn.cursor() as cur:
        cur.execute(FINISH_SQL, {
            "id": job_id, "worker": worker,
            "state": state, "result": result, "error": error,
        })
        # No row returned means the lease was reclaimed; discard our result.
        return cur.fetchone() is not None

def run(worker):
    with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
        conn.autocommit = True
        while True:
            job = claim(conn, worker)
            if job is None:
                time.sleep(1)
                continue
            job_id, attempt, payload = job
            try:
                resp = client.chat.completions.create(
                    model=payload["model"],
                    messages=payload["messages"],
                    timeout=600.0,
                )
                text = resp.choices[0].message.content
                finish(conn, worker, job_id, "succeeded", result=text)
            except Exception as exc:
                state = "failed" if attempt >= MAX_ATTEMPTS else "queued"
                finish(conn, worker, job_id, state, error=str(exc))

if __name__ == "__main__":
    run(f"worker-{uuid.uuid4()}")

The same lease concept as SQL, which is also what a reaper runs on a timer:

-- Reclaim jobs whose worker stopped heartbeating, while attempts remain.
UPDATE jobs
   SET state = 'queued',
       lease_owner = NULL,
       lease_expires_at = NULL
 WHERE state = 'running'
   AND lease_expires_at < now()
   AND attempt < 3;

-- Give up on jobs that exhausted their attempts.
UPDATE jobs
   SET state = 'failed',
       error = COALESCE(error, 'lease expired after max attempts')
 WHERE state = 'running'
   AND lease_expires_at < now()
   AND attempt >= 3;

Webhook verification in practice

The handler below is deliberately boring: verify, deduplicate, enqueue, acknowledge. The unusual detail is that it works on the raw request buffer, which is what makes the signature check correct.

import crypto from "node:crypto";
import type { Request, Response } from "express";
import { pool } from "./db";
import { enqueue } from "./queue";

const TOLERANCE_SECONDS = 300;

function verify(rawBody: Buffer, header: string, secret: string): boolean {
  const parts = Object.fromEntries(
    header.split(",").map((kv) => kv.split("=") as [string, string]),
  );

  const timestamp = Number(parts.t);
  if (!Number.isFinite(timestamp)) return false;
  if (Math.abs(Date.now() / 1000 - timestamp) > TOLERANCE_SECONDS) return false;

  const expected = crypto
    .createHmac("sha256", secret)
    .update(`${parts.t}.`)
    .update(rawBody)
    .digest("hex");

  const a = Buffer.from(expected);
  const b = Buffer.from(parts.v1 ?? "");
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

// Mount with express.raw({ type: "application/json" }) so req.body is a Buffer.
export async function webhookHandler(req: Request, res: Response) {
  const raw = req.body as Buffer;
  const header = req.header("x-provider-signature") ?? "";

  if (!verify(raw, header, process.env.WEBHOOK_SECRET ?? "")) {
    res.status(401).end();
    return;
  }

  const event = JSON.parse(raw.toString("utf8"));

  const inserted = await pool.query(
    `INSERT INTO webhook_events (event_id, received_at)
     VALUES ($1, now()) ON CONFLICT (event_id) DO NOTHING
     RETURNING event_id`,
    [event.id],
  );

  // Duplicate delivery: acknowledge without reprocessing.
  if (inserted.rowCount === 0) {
    res.status(200).end();
    return;
  }

  await enqueue("job-results", { eventId: event.id, payload: event.data });
  res.status(200).end();
}

Operational concerns

The stuck-job reaper

Run the reaper every 30 to 60 seconds and make it idempotent, so a double run is harmless. Alert when one pass reclaims more than a small threshold: a burst means workers are crashing in bulk, an infrastructure problem rather than bad luck.

Alert on queue age, not just depth

Queue depth alone is misleading. Ten thousand pending jobs is comfortable at 50 milliseconds each and catastrophic at five minutes each. Alert on the age of the oldest unclaimed message and the p95 time from queued to running.

Per-tenant fairness

One tenant submitting 100,000 jobs must not starve everyone else. The cheapest control is a per-tenant concurrency cap enforced at claim time, so no tenant holds more than N worker slots. Weighted fair queueing is the fuller answer: order candidates by how far each tenant sits below its fair share.

Backpressure when the provider is slow

When the provider returns 429s or latency climbs, the instinct is to retry harder. That is backwards. Slow the consumers, honour Retry-After, and trip a circuit breaker after consecutive failures. Bound the queue and reject new submissions with 429 once full: refusing a job immediately beats accepting one you cannot finish.

When webhooks are worth it

My default is submit-and-poll with a reconciliation worker, and webhooks only once volume or latency makes polling genuinely expensive. Polling has no failure mode you have not already handled: your status endpoint is the source of truth, and a client that goes away leaves no dangling state.

Webhooks invert that trade. You gain delivery latency measured in milliseconds instead of your poll interval, and you pay with a public endpoint, HMAC verification, a replay window, deduplication, retry handling, and a reconciliation sweep anyway.

They are worth it when the job is user-facing and long, when you deliver tens of thousands of completions per day, or when the consumer is a server you control. A gateway that normalises providers behind one endpoint lets you start with polling and add webhooks later without changing your client contract. Qora API is one option: one OpenAI-compatible key across GPT, Claude, Gemini and others, so your delivery logic survives a provider swap.

Frequently asked questions

How fast should a webhook endpoint respond?

Under five seconds, and ideally under 500 milliseconds. Verify the signature, insert the event id under a unique constraint, enqueue, return 200. Anything else belongs in a worker. A handler that occasionally takes 8 seconds against a 10-second delivery timeout accumulates retries during spikes, precisely when you can least afford them.

Can I get exactly-once delivery?

No, not end to end. At-least-once transport plus idempotent consumers is the achievable target, and it yields exactly-once effects as long as every side effect is keyed. Treat any claim of exactly-once delivery spanning a queue, a database and a third-party API as a description of deduplication.

What if a provider never sends the webhook?

You must detect it. Record the expected delivery deadline at submission and run a sweep for anything past it. If the provider reports the job finished, settle it locally and emit the event yourself. Without the sweep, a webhook lost during a deploy becomes a job stuck forever.

Should I use streaming or webhooks?

They solve different problems. Streaming optimises time-to-first-token for a user watching output appear; webhooks optimise completion notification for a job nobody is watching. A long document analysis might stream partial sections and still fire a webhook when the structured result is ready.

How do I choose a lease length?

Start from your p99 job duration and add margin, then add a heartbeat so the value need not cover worst-case runtimes. A 60-second lease with a 15-second heartbeat handles jobs of any length while keeping reclaim latency under a minute. Sizing the timeout to the worst case means a crashed worker blocks that job for its full duration.

Conclusion

Asynchronous delivery is not a feature you bolt onto an API; it is a different contract. Once submission and completion are separated you own a durable job row, a lease, a retry policy, an idempotency story and a reconciliation loop. In return, a 90-second generation stops being a failure and becomes ordinary.

Build the state machine first, get the lease and idempotency keys right, and start with polling. Add webhooks when you can point at a specific latency or cost problem that polling causes. Keep the reconciliation sweep running either way, because it turns an at-least-once world into a correct one. A unified gateway such as Qora API reduces that provider-specific surface to one endpoint and one billing model.

Related reading

• Batch AI APIs: Processing Millions of Requests Affordably
• How to Handle AI API Rate Limits and 429 Errors
• Building Reliable AI Agents: Guardrails, Retries, and Human-in-the-Loop
• Idempotency and Safe Retries for AI APIs

Self-hosting an AI gateway means running the proxy layer that sits between your applications and model providers on infrastructure you control. It buys you credential isolation, routing logic you can change in a single deploy, and per-request cost attribution that a black-box endpoint will never give you. It costs an on-call rotation, a provider schema-drift treadmill, and a stateful data plane that must not fall over. Below a few hundred million tokens a month, a managed gateway is usually the better trade; above that, or under a jurisdiction constraint you cannot negotiate away, self-hosting starts to win.

What a managed gateway gives you, and what it structurally cannot

A managed gateway – qoraapi.com being one example – collapses a lot of undifferentiated work into a single OpenAI-compatible endpoint: credentials handled by someone else, failover already wired, one invoice, one rate-limit surface, one place to look when a provider degrades. That is not marketing. If your routing policy is “use model X, and if it 429s use model Y,” rebuilding it buys you nothing.

The gaps appear when requirements stop being generic. A managed gateway cannot guarantee that a prompt body never leaves a specific region, because the payload is by definition in someone else’s process. It cannot route on your tenant taxonomy – enterprise on EU inference, free tier on the cheapest healthy provider, anything flagged medical pinned to a zero-retention endpoint. It cannot redact PII before egress or cache on your own key taxonomy, and it cannot let you survive a provider relationship ending without an application release.

Those are the four reasons teams self-host: data control, bespoke routing, cost transparency, and provider independence. Everything else is a proxy with extra steps.

The real cost of ownership

The infrastructure is the cheap part. The expensive part is that a gateway is a Tier-0 dependency for every LLM feature you ship, and it changes underneath you continuously: providers add parameters, deprecate parameters, change defaults, and quietly alter streaming chunk shapes. Adapter code churns constantly even when your own product is frozen.

Steady state for a moderately complex gateway is a quarter to a half of an engineer’s time, permanently – incident response, provider migrations, credential rotation, capacity work, config reviews. The first quarter costs more, and any provider migration costs more again. Teams that budget for the build and not the maintenance end up with a half-maintained gateway, which is worse than a managed one: it fails in ways nobody on the current team understands.

The component map

Every gateway decomposes into the same ten parts. Knowing which ones you are skipping is the difference between a working proxy and an outage.

• Credential vault. Provider keys must never live in application config. Store them with envelope encryption under a KMS key, scoped per environment, rotated on a schedule you have rehearsed. The important property is asymmetry: applications hold a gateway key, the gateway holds provider keys, so a leaked application key is revoked without touching provider credentials. See the AI API security checklist.
• Provider adapters. One module per provider, translating your canonical request into its native shape. This is the highest-churn code in the system, so test it hardest.
• Request normalisation. A canonical schema. OpenAI’s chat completions shape is the de facto lingua franca, with well-known leaks around tool calls and multimodal content.
• Routing policy. Rules over model alias, tenant, region, cost ceiling and provider health.
• Rate limiting. Per key, per tenant, per model. Token-aware, not just request-count-aware, or one long-context request bypasses your protection.
• Retry and failover. Error classification, retry budgets, jitter, and a circuit breaker per provider.
• Response cache. Exact-match at minimum; semantic if the workload justifies it.
• Usage metering. Token counts, cost and latency attributed to a tenant and a feature.
• Observability. Per-request traces with prompt and response redaction, plus aggregate metrics.
• Admin API. Keys, budgets, config versions, model catalogue. The control plane.

The split that matters most is control plane versus data plane. The control plane can be down for an hour and nothing breaks, provided the data plane serves from its last-known-good configuration. Build that property in on day one; retrofitting it during an incident is not possible.

Deployment topologies compared

Topology	Added latency	Blast radius	Upgrade path	Where it fits
Central cluster	One in-region network hop plus TLS and proxy work	Everything that talks to the gateway	One deploy changes behaviour fleet-wide, instantly	The default. Fix a provider outage once for all consumers.
Sidecar per pod	Loopback only	One pod, but a bad config needs every pod rolled to fix	Roll every workload; config distribution becomes the hard problem	Strong per-tenant network isolation requirements in a Kubernetes-heavy shop
In-process library or SDK	None	One process	Every application redeploys, in every language	Single-language monoculture, one owning team, no cross-team consumers

Default to a central cluster. It is the only topology where a routing fix, a credential rotation or a provider failover lands everywhere at once. Sidecars sound appealing until you realise per-pod connection pools multiply your provider connection count by your replica count – exactly the thing providers rate-limit you on. In-process libraries work for a single-language shop, but you reimplement config, metering and observability once per language, and you lose failover without an application release – the main reason you built a gateway.

Stateless versus stateful: keeping counters out of the request path

The request path should be stateless wherever it can be. Four things genuinely cannot be, and each needs a deliberate failure policy rather than a default.

Stateful component	Consistency needed	Sensible failure policy
Rate-limit counters	Strong per key, approximate globally	Fail open, with alerting. Rejecting all traffic because Redis blipped is worse than a short burst of overspend.
Budget and spend counters	Strong if the cap is contractual, eventual if it is advisory	Fail closed only when the number is a hard quota. Batch increments for advisory counters and accept bounded overshoot.
Response cache	None	A miss is the normal path. A cache outage must degrade to full origin traffic, never to 5xx.
Circuit-breaker health	Per instance is enough	Local state, converges in seconds. Never make it a shared dependency.

The discipline is the same in all four cases: bound the round trip. A rate-limit check that can block for two seconds is worse than no rate limiter, because it converts a load problem into a fleet-wide latency problem. Give the counter a hard timeout in the tens of milliseconds and decide in advance what happens when it expires.

The check must be one atomic round trip: read-modify-write from application code races under concurrency and lets bursts through.

-- token bucket, atomic in one round trip, no read-modify-write race
-- KEYS[1] = bucket key   ARGV = capacity, refill_per_sec, now_ms, cost
local capacity = tonumber(ARGV[1])
local refill   = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local cost     = tonumber(ARGV[4])

local b      = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(b[1]) or capacity
local ts     = tonumber(b[2]) or now

tokens = math.min(capacity, tokens + (now - ts) / 1000 * refill)

if tokens < cost then
  redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
  redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / refill * 1000))
  -- second return value is the retry-after hint in milliseconds
  return { 0, math.ceil((cost - tokens) / refill * 1000) }
end

redis.call("HMSET", KEYS[1], "tokens", tokens - cost, "ts", now)
redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / refill * 1000))
return { 1, 0 }

A routing policy you can actually run

Policy belongs in declarative config, versioned and validated in CI, not in code branches. A workable shape:

# gateway/policies/chat-default.yaml
version: 3
route: chat-default

match:
  model_alias: [ "chat-fast", "chat-balanced" ]

providers:
  - id: openai-primary
    adapter: openai
    model: gpt-4.1-mini
    weight: 100
    timeout:
      connect_ms: 400
      first_byte_ms: 2500
      total_ms: 30000

  - id: anthropic-fallback
    adapter: anthropic
    model: claude-sonnet-4
    weight: 0                 # failover target only, never load-balanced
    timeout:
      connect_ms: 400
      first_byte_ms: 3000
      total_ms: 45000

  - id: bedrock-eu
    adapter: bedrock
    region: eu-central-1
    model: meta.llama3-70b
    weight: 0
    timeout:
      connect_ms: 600
      first_byte_ms: 4000
      total_ms: 60000

retry:
  max_attempts: 3
  budget_ratio: 0.10          # retries may never exceed 10% of total traffic
  backoff: exponential_jitter
  retry_on: [ "connect_timeout", "first_byte_timeout", "http_429", "http_5xx" ]
  never_retry_on: [ "http_400", "http_401", "http_403", "content_filter" ]

circuit_breaker:
  error_rate_threshold: 0.50
  min_requests: 20
  open_seconds: 15

cache:
  exact_match: true
  ttl_seconds: 3600
  key: [ model_alias, messages, temperature, tenant_id ]

limits:
  tenant_rpm: 600
  tenant_tpm: 400000

Two details carry most of the value. First, budget_ratio: without it, retries are unbounded amplification, as documented in the multi-provider failover playbook. Second, weight: 0 on the fallbacks – a fallback receiving steady traffic is not a fallback but a second primary with worse cost characteristics and no warm-up guarantee.

Scaling the data plane

Connection pooling is the whole game. Keepalive pools must be sized to peak concurrency, not to instance count, and each provider needs its own pool because their latency and rate-limit profiles differ. HTTP/2 multiplexing helps, but a single connection carries a bounded number of concurrent streams – typically around a hundred – and beyond that requests queue invisibly. An undersized pool shows up as p99 latency that looks like provider slowness but is local queueing.

Streaming changes the capacity model. An SSE response holds a connection for the entire generation. An instance serving five hundred concurrent streams is nearly idle on CPU and bounded by memory buffers and file descriptors, not compute. Request-per-second is therefore the wrong autoscaling signal; scale on in-flight streams. Disable response buffering at every hop – proxy_buffering off in nginx, X-Accel-Buffering: no from the proxy – or your streaming endpoint will deliver the whole answer in one burst after a ten-second stall. Never compress an event stream.

import asyncio, time, httpx
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

UPSTREAM = {
    "openai":    "https://api.openai.com/v1/chat/completions",
    "anthropic": "https://api.anthropic.com/v1/messages",
}

# One pool per provider. pool=0.1 is the admission-control valve: if no
# connection is free within 100ms we shed load instead of queueing forever.
POOLS = {
    "openai": httpx.AsyncClient(
        limits=httpx.Limits(max_connections=256, max_keepalive_connections=128),
        timeout=httpx.Timeout(connect=0.4, read=30.0, write=5.0, pool=0.1),
    ),
    "anthropic": httpx.AsyncClient(
        limits=httpx.Limits(max_connections=128, max_keepalive_connections=64),
        timeout=httpx.Timeout(connect=0.4, read=45.0, write=5.0, pool=0.1),
    ),
}

app = FastAPI()

@app.post("/v1/chat/completions")
async def chat(request: Request):
    body     = await request.json()
    tenant   = request.headers["x-tenant-id"]
    provider = route(body)          # policy lookup, no I/O

    if not body.get("stream"):
        return await collect(provider, body, tenant)

    async def relay():
        started = time.monotonic()
        client  = POOLS[provider]
        async with client.stream(
            "POST", UPSTREAM[provider], json=body,
            headers={"Authorization": "Bearer " + await keyring.get(provider)},
        ) as up:
            up.raise_for_status()
            async for chunk in up.aiter_raw():
                if await request.is_disconnected():
                    break            # stop paying for tokens nobody will read
                yield chunk
        # fire-and-forget: metering must never sit on the response path
        asyncio.create_task(emit_usage(tenant, provider, time.monotonic() - started))

    return StreamingResponse(
        relay(),
        media_type="text/event-stream",
        headers={"cache-control": "no-store", "x-accel-buffering": "no"},
    )

The is_disconnected check is not an optimisation. Without it, a user closing a tab leaves the upstream generation running, and you pay for every token of it.

The sticky-session trap. The temptation with long-lived streams is to pin a client to an instance by cookie or source IP. Do not. Stickiness destroys load balancing, strands in-flight streams on instances you are draining, and turns every deploy into a rolling brownout. The alternative is a stateless router plus graceful shutdown: on SIGTERM stop accepting new connections, let in-flight streams finish up to a deadline, then exit. Set the load balancer idle timeout above your maximum generation time, or long streams get severed mid-sentence by a healthy-looking proxy.

Backpressure has three layers, and you need all three: connection limits at the load balancer, a concurrency semaphore per instance, and a per-tenant concurrency cap. When saturated, reject with 429 and a Retry-After rather than queueing: queueing converts a capacity problem into a timeout storm, and timeouts consume capacity while producing nothing. The pool=0.1 timeout above is the mechanism – deliberate load shedding, not a misconfiguration.

Failure modes, in the order they will bite you

1. Config rollout that breaks all traffic. The most common self-inflicted gateway outage by a wide margin: one inverted condition, one typo in a provider ID, or a schema change that silently drops a field, and 100% of traffic fails at once. Validate config against a strict schema in CI and reject unknown fields; roll out 1% then 10% then 100% with automatic rollback on error-rate delta; and keep last-known-good config in the data plane so a control-plane failure cannot take down serving.
2. Retry storms. A provider starts 429ing, every instance retries, the retries multiply load threefold, the provider pushes back harder, and a transient degradation becomes a sustained outage. Fixes: a retry budget as a fraction of total traffic, full-jitter backoff, a per-provider circuit breaker, and a hard rule never to retry on 400, 401, 403 or content-filter responses – those are deterministic and will fail identically three times.
3. Cache stampede. A popular prompt’s entry expires, two hundred concurrent requests miss simultaneously, and all two hundred hit the provider with an identical payload. Use single-flight: the first request fills the entry, the rest await the same future. Add TTL jitter so entries do not expire in lockstep, and a stale-while-revalidate window so a refresh never blocks a reader.
4. Hot partitions in the rate limiter. One tenant with a shared bucket saturates a Redis shard and adds latency to every other tenant on it, because a single global counter is a single hot shard. Shard by tenant, give any tenant large enough to saturate one shard its own bucket with a longer refill window, and monitor per-shard latency rather than the aggregate.
5. Provider schema drift. A provider adds a response field, changes a default, or deprecates a parameter, and your adapter silently drops it – behaviour changes without an error. Use contract tests against recorded fixtures per provider version, count unrecognised response fields and alert when the counter moves, and run strict mode in staging that errors on unknown fields.

Notice what is absent from this list: provider downtime. Providers are more reliable than the code you write on top of them, and the outages you remember will be your own.

A worked TCO example

An illustrative calculation with visible arithmetic, not a benchmark or a quote. Substitute your own numbers; the shape of the result is what matters.

• Volume. 5,000,000 requests/day. Mean request payload 4 KB, mean response payload 8 KB, so 12 KB per request.
• Peak concurrency. Mean rate is 5,000,000 / 86,400 = 58 requests/second. With a 3x diurnal peak that is 174 requests/second. If 30% of traffic streams for a mean 8 seconds, peak concurrent streams are 174 x 0.30 x 8 = 418.
• Compute. At 250 concurrent streams per instance for headroom, 418 / 250 = 1.7, so 2 instances for peak plus 1 for AZ and deploy redundancy = 3 instances. A 4 vCPU / 8 GB instance at $0.19/hour is 0.19 x 730 = $139/month. Three of them: $417/month.
• Egress. 5,000,000 x 12 KB = 60 GB/day, or 1,800 GB/month. At $0.09/GB: $162/month.
• Load balancer and NAT data processing: $65/month.
• State store. A managed Redis with a replica: $80/month.
• Observability. Full trace capture would be 5,000,000 x 1.5 KB = 7.5 GB/day. Sample 10% instead: 750 MB/day, about 22 GB/month, roughly $7 at $0.30/GB ingest, plus $60 for metrics and logs. Call it $70/month.
• Infrastructure subtotal: 417 + 162 + 65 + 80 + 70 = $794/month.
• Engineering. 0.35 FTE at $200,000 fully loaded = $70,000/year = $5,833/month.
• Total: $794 + $5,833 = $6,627/month, or $0.044 per 1,000 requests at 150,000,000 requests/month.

The ratio is the point: infrastructure is roughly one seventh of the cost and people are six sevenths. Two consequences follow. First, “it is cheaper” is not a valid argument for self-hosting – the dominant term barely moves between 5M and 50M requests/day, so per-request economics improve sharply with volume while absolute cost hardly changes. Second, compare the fully loaded $6,627 against what a managed gateway costs at this volume. If the managed fee lands in the same range, self-hosting is a bad trade on cash terms alone.

When self-hosting is worth it, and when it is not

Self-host when a regulator or customer contract requires payloads to stay inside a specific VPC or jurisdiction and no vendor will commit to that contractually. When routing must be genuinely bespoke – cost-aware tiering, per-tenant provider pinning, pre-egress redaction, region-pinned inference for a subset of traffic. When you need to switch providers without an application release, which is an architectural property, not a cost one.

Do not self-host when you are pre-product-market-fit and routing is “one provider with occasional failover.” When nobody will be paged for a proxy at 3am. When you cannot commit a quarter of an engineer’s time permanently, because a half-maintained gateway is worse than a managed one – it fails in ways nobody currently employed understands. And when the only motivation is cost, which the arithmetic above shows rarely survives contact with the staffing number.

The strongest position is not to pick one. Keep your application speaking a single canonical, OpenAI-compatible schema, put credentials and routing behind an interface you own, and treat the gateway as a swappable component. Start managed, and migrate when a concrete requirement – not a vibe – forces you. Teams that self-hosted for a compliance obligation are happy with the decision; teams that self-hosted to save money usually are not. For a wider survey of the managed option, see the AI API gateway guide.

Frequently asked questions

How much latency does a self-hosted gateway actually add?

One in-region network hop plus TLS termination and proxy work. Provider time-to-first-token dominates the end-to-end number, so if proxy overhead is more than a small fraction of total TTFB, the cause is almost always a state lookup on the request path: an unbounded Redis call, a synchronous config fetch, or uncached DNS resolution. Instrument the proxy span separately from the upstream span; if you cannot separate them, you have an observability problem before a latency problem.

Do I need three instances, or is two enough?

Two instances survive a single failure but run at 50% capacity each at peak, so you pay for redundancy you can never use. Three runs each at 67% and absorbs both an AZ loss and a rolling deploy. If your gateway fronts a chat feature, two is defensible; anything with a revenue SLA needs three.

Should the response cache live in the gateway or the application?

Exact-match caching belongs in the gateway: the key is derivable from the request and the benefit is shared across consumers. Semantic caching is a product decision – the embedding model, similarity threshold and staleness window depend on your workload – so it belongs where those can be tuned per feature. The mechanics are covered in the semantic caching writeup. Either way, a cache miss must be an ordinary code path, never an error.

How do I handle provider-specific parameters the canonical schema does not have?

An explicit, allowlisted passthrough bag – something like provider_options – scoped to a single request. Never a blanket passthrough of arbitrary fields: it lets application code set parameters that violate your routing invariants, and it is how you end up with a request pinning a model the router thought it was choosing. The allowlist is small and grows by review, which is the point.

What should I instrument first?

Four things, in order: a per-request trace carrying provider, model, attempt number and outcome; upstream time-to-first-byte and total duration as separate histograms; prompt and completion token counts attributed to tenant; and a retry counter broken down by reason. Those answer almost every question you will have during an incident. Prompt and response bodies are useful in staging and dangerous in production – see the observability guide for keeping the useful parts while keeping payloads out of your log store.

Conclusion

Self-hosting an AI gateway is a straightforward engineering problem wrapped in a hard organisational one. The engineering is well understood: normalise requests, isolate credentials, keep the request path stateless, bound every state lookup, control retries with a budget, and make the control plane incapable of taking down the data plane. The hard part is that you are signing up for permanent ownership of a Tier-0 service whose dependencies change every few weeks.

Decide on requirements, not on cost. If a jurisdiction constraint, a bespoke routing rule or a genuine provider-independence requirement forces your hand, self-host and staff it properly – a quarter of an engineer, permanently, not a sprint. If none apply, run managed, keep your client interface canonical, and preserve your ability to move later. That optionality is worth more than the infrastructure savings, and it is the one decision you cannot retrofit cheaply once application code has learned a vendor’s quirks.

Related reading

• What Is an AI API Gateway? A Practical Guide for Developers
• Ultimate Guide: How to Choose the Best AI API Gateway in 2026
• AI Gateway vs API Gateway: Key Differences and When to Use Each
• How to Build a Multi-Provider AI Failover Layer for 99.9% Uptime

The bills that hurt are never the ones you planned for. A workload running at 200 USD a day does not become a 20,000 USD day because a model got more expensive; it becomes one because something looped, retried, or was invoked ten thousand times by a code path nobody was watching. Guardrails are a different problem from cost optimisation: optimisation lowers the baseline, while budget caps, kill switches and anomaly alerts bound the worst case.

Optimisation is not a control

Caching, model routing and prompt trimming reduce the price of a normal request. None of them stops an abnormal one. A 70 percent cache hit rate is a great number until an agent loop generates a unique prompt on every iteration and pays full price forever. Model routing does nothing when the runaway path legitimately needs the frontier model.

“Our cost per request is down 40 percent” and “we cannot spend more than 5,000 USD this month” are unrelated claims. The first is a cost reduction exercise; the second is admission control, and it must be enforced in the request path rather than observed afterwards in a dashboard.

Six failure modes that actually cause runaway bills

Runaway spend comes from a handful of structural bugs, each with an early signature and a cheap control.

An agent loop that never terminates. The model calls a tool, the tool returns something unusable, and the model calls the tool again. Nothing is broken in the traditional sense, so nothing alerts, and the cost curve is worse than linear because context grows each turn. Worked example: a 2,000-token system prompt plus 1,500 tokens per turn means input at turn n is 2,000 + 1,500(n-1). Over 200 turns that is 200 × 2,000 + 1,500 × (200 × 199 / 2) ≈ 30.25M input tokens, which at 3 USD per million is 90.75 USD for one conversation. A job fanning out 5,000 of them is 453,750 USD. The missing control is a maximum-turn and per-run token budget.

A retry storm. A retry wrapper with a short timeout, a reset backoff counter, or missing jitter can multiply one logical request into hundreds. Fifty workers retrying ten times a second against a failing call is 500 requests per second; at 0.02 USD each, roughly 36,000 USD per hour. Errors are cheap only if they fail fast — a call that burns input tokens before returning a 500 costs full price every attempt. Idempotency keys and a retry budget fix it; see the retry and 429 guide.

A user-triggered bulk action with no per-user cap. Someone selects “all 400,000 records” in a UI that fans out one model call per record. The org budget is fine; the individual account is not.

A runaway eval or backfill job. A scheduled job re-scores the last 90 days of traffic against a new prompt. It never appears in latency dashboards and runs at 02:00. Backfills cause more surprising months than anything else: one-off code, written quickly, with no budget.

A webhook redelivery loop. Your handler returns a 500 after the model call succeeds, the provider redelivers, and your handler calls the model again. Every redelivery is paid inference, and the signature is a burst of identical idempotency keys.

A prompt that grows the context window every turn. The slow burn: each turn appends, nothing is evicted, and per-request cost creeps up a few percent per turn. No per-request limit ever trips.

Failure mode	Early signal	Control that stops it
Non-terminating agent loop	Turns per run climbing; tokens per run rising superlinearly	Max-turn cap, per-run token budget, spend-rate alert
Retry storm	Retries per logical request; requests per second per client; 5xx ratio	Retry token bucket, exponential backoff with full jitter, idempotency keys
Uncapped bulk action	Requests per user per minute; spend per user	Per-user daily cap, job-size ceiling, async queue with confirmation
Runaway eval or backfill	Off-hours spend; spend grouped by feature tag	Dedicated budget per feature, dry-run sample, hard job ceiling
Webhook redelivery loop	Duplicate idempotency keys; requests from one source per minute	Idempotency keys with a dedupe window, per-source rate cap
Growing context window	Tokens per request z-score; input-to-output ratio	Context compaction, sliding window, per-request token ceiling

Budget hierarchy: put each control at the level that owns the failure

A single org-level budget is a smoke detector in a warehouse. It tells you the building is on fire and nothing about which aisle. Each level should exist to catch a failure the level above cannot localise.

Level	Typical control	Why it belongs here
Org	Monthly hard ceiling, global kill switch, billing alerts	Last line of defence; the number finance signs off on
Team	Monthly budget with a named owner	Accountability without blocking unrelated teams
Environment	Separate budgets; development and staging capped hard	Stops an experiment or load test eating production headroom
User	Daily cap plus per-minute rate limit	Catches one account’s loop or bulk action while everyone else works
Feature	Budget per feature tag, e.g. summarise_v3	Catches a bad deploy of one code path before it becomes an org incident
Request	max_tokens ceiling, context limit, pre-flight reservation	Bounds the worst possible single call

Per-user caps catch what org caps structurally cannot. When an org budget trips, every user is blocked, including the 99.9 percent who behaved correctly, and the response starts with “who did this” — an investigation that takes hours when requests are untagged. A per-user cap turns the same event into one account hitting a limit. Enforce at the narrowest scope that covers the blast radius, and keep the wider scope as a backstop.

Soft caps, hard caps, and the 70/90/100 ladder

A hard cap with no warning is bad engineering: the first signal a customer gets is total failure, at the worst possible time, with no chance to react. A soft cap with no enforcement is not a control, just a notification people learn to filter. You need a ladder that both warns and enforces, applied per scope.

• 70 percent — warn. Alert the owning team, not the on-call rotation. Nothing is blocked. Increase trace sampling for this scope and put the top spenders by feature in the alert body.
• 90 percent — degrade. Shed cost automatically: route eligible traffic to a cheaper model, disable optional enrichment, tighten max_tokens, stop accepting batch work. Users still get answers; they get smaller ones.
• 100 percent — block. Reject new requests for that scope with a machine-readable error carrying the scope, the reset time, and a link to request an increase. In-flight requests finish; killing them mid-stream wastes paid tokens.
• 120 percent — kill. If spend still passes 120 percent of a hard limit, something bypassed the reservation path. Trip the kill switch and page a human: reaching this tier is a bug, not a budget decision.

One knob per scope, one owner, one place to look when behaviour changes.

Pre-flight estimation, atomic reservation, and reconciliation

You cannot enforce a budget you only learn about after the call. Every request needs a cost estimate before dispatch, a reservation against the relevant scopes, and a settlement once real usage is known.

Estimating before you call

Use the model’s real tokeniser when the request is large enough to matter, and a heuristic when it is not. For most chat traffic a character-based estimate lands within about 15 percent, which is fine for reservation as long as you reconcile: roughly four characters per token for Latin text, one and a half to two for CJK, plus 20 to 30 tokens of chat framing. Then price it using the output cap, not a hopeful average: estimate = input_tokens × price_in + max_tokens × price_out.

Reserving against max_tokens deliberately over-reserves, which is the correct default: over-reservation costs headroom, under-reservation costs the guarantee. If callers set max_tokens to 4096 and use 300, tighten the callers rather than weakening the guard.

The concurrency race

The naive implementation reads the remaining budget, compares it to the estimate, and writes the new total. Two requests arriving in the same millisecond both read 1.00 USD of headroom, both estimate 0.80 USD, both pass, and both spend — a 60 percent overrun from two concurrent calls, unbounded at real concurrency. This lost-update race is why homegrown guards fail precisely when traffic spikes.

Two fixes exist. An atomic counter performs the check and the increment in one operation, via a Redis Lua script or a conditional SQL update. A reservation ledger writes a row per request in a reserved state and moves it to settled afterwards. The counter is faster; the ledger gives attribution and post-hoc reconciliation. Ship both if you can, with the ledger as source of truth.

Where to enforce: application, gateway, or both

Application-level checks are the only place with enough context to know which user, feature and logical operation a request belongs to, and the easiest layer to bypass. Every new service, notebook, cron job and script is a fresh chance to call the provider without the guard.

A gateway sits in front of every provider call by construction and cannot be bypassed by new code as long as the credentials live behind it. That is the operational reason never to distribute raw provider keys: issue gateway keys with per-key budgets. The gateway knows the model, the token counts and the account, so it enforces org and tenant ceilings, rate limits and kill switches without caller cooperation.

Its blind spot is intent: it does not know this call is the 40th turn of a loop, or that it belongs to a backfill that should never exceed 200 USD. So the application should enforce semantic caps and pass a pre-flight estimate, while the gateway enforces the hard ceiling, the rate limit and the kill switch, and rejects requests arriving without a valid estimate header. That is what a gateway such as Qora API is built for: one key across providers with per-key budgets and unified metering.

Kill switches that are fast, safe, and rehearsed

A kill switch is a manual, deliberate stop. Unlike a circuit breaker, which is automatic and trips on error rate, a kill switch trips on money. It needs three properties.

Scoped. Support at least global, per-tenant, per-model and per-feature scopes. Stopping one tenant is a routine action; stopping everything is a company-level decision. If the only switch you have is global, nobody will pull it, which means it does not exist. The switch should stop new requests and let in-flight ones finish — except streaming calls, whose cost is unbounded until they end.

Fast. Read the flag from a cache with a short TTL, in the request path — not from a config deploy, not from a database query. A switch that takes four minutes to propagate arrives after the spend. Target seconds from flip to effect, and measure it.

Testable in production without a real incident. This is the property everyone skips and the one that decides whether the switch works when you need it. Run it in shadow mode — flip it for one internal tenant or one percent of traffic — and confirm callers get the expected error, retries do not amplify the rejection, and the alert fires. Do it on a schedule, because an untested kill switch is a boolean never evaluated under load.

Anomaly detection that catches the burn before the invoice

A per-request threshold misses slow burns by construction, and a daily alert arrives after the money is gone. Evaluate on minute-level aggregates, and prefer rules you actually ship.

• Spend rate versus trailing baseline. Compare spend in the last ten minutes against the median of the same bucket over the previous seven days. Alert when the ratio exceeds five and the value clears a floor, so 0.02 USD to 0.10 USD does not page anyone at 4am.
• Tokens per request z-score. Track mean and standard deviation of input tokens per feature over a rolling window; alert at mean plus three standard deviations. This catches the growing context window, which no cost-per-request rule sees.
• Requests per user per minute. Humans do not make 400 requests a minute; loops and bulk actions do.
• Cache hit-rate collapse. A drop from 60 percent to 5 percent means a cache-key bug or a prompt change that invalidated every entry. Both cost money and are invisible in latency metrics.
• Model-mix shift. Alert when the share of requests on your most expensive model moves by more than a few points. A routing bug sending everything to the frontier model is silent and expensive.

Alert on the derivative, not the level: a scope that always spends 500 USD a day should not page anyone for spending 500 USD a day. Put the top three spenders by tag in the alert body, because “spend is up 6x, and 92 percent of it is tenant_4471” ends an investigation that “spend is up 6x” starts. These are the aggregates described in the observability stack.

Graceful degradation beats hard failure

Blocking is the last resort. Most budget pressure can be absorbed by changing what the system does rather than whether it responds.

Degradation	User impact	When to use
Fall back to a cheaper model	Lower quality on hard inputs, still correct on easy ones	Classification, extraction, summarisation, and any task where a cheap model passes your eval set
Disable an optional step	Less rich answer; nothing promised is missing	Enrichment such as reranking, second-opinion passes, speculative tool calls
Shorten the context	Long-range detail may be lost; recent turns preserved	When context is the cost driver and recency is what users rely on
Queue for later	Latency moves from seconds to hours	Non-interactive work: evals, backfills, report generation, embedding refreshes
Return a partial answer with a notice	Incomplete but honest and immediate	When the user prefers something now over everything later
Reject with a clear error	Work blocked until the budget resets or is raised	Only when the alternative is unbudgeted spend

The routing decision must be precomputed, not improvised at 90 percent budget. Which tasks have an acceptable cheap-model fallback is a quality question that belongs in an eval run, not an incident. The model routing guide covers how to establish that mapping.

Attribution: tag every request or debug forever

Without tags, a spend spike is an unbounded investigation: a number, a timestamp, and hours of grepping logs for the code path that changed. With tags it is one query. Every request should carry tenant, user, feature, environment, model, and a trace identifier propagated through retries and tool calls, so a storm of 400 provider calls collapses into one logical request.

The trace identifier is the piece teams most often omit and the one that makes retry storms legible: without it, a storm looks like a traffic increase. The feature tag is the other high-value dimension because it maps onto a deploy, and GROUP BY feature ORDER BY usd DESC answers “what shipped” in seconds. The metering model behind this is described in the usage metering post.

A concrete implementation: reserve, then reconcile

The guard below estimates cost from the prompt and the output cap, reserves atomically against a scope, and reconciles with actual usage when the call returns. Atomicity comes from a Redis Lua script, so the read-check-write sequence cannot interleave.

import uuid
from dataclasses import dataclass
from decimal import Decimal

import redis

# Check and increment happen in one atomic step. Without this, two concurrent
# requests both read the same headroom and both pass.
RESERVE_LUA = """
local key = KEYS[1]
local amount = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local ttl = tonumber(ARGV[3])
local current = tonumber(redis.call('GET', key) or '0')
if current + amount > limit then
  return {'-1', tostring(limit - current)}
end
local next_total = current + amount
redis.call('SET', key, next_total, 'EX', ttl)
return {tostring(next_total), tostring(limit - next_total)}
"""

@dataclass(frozen=True)
class Reservation:
    request_id: str
    scope: str
    period: str
    amount_usd: Decimal

class BudgetExceeded(Exception):
    """Raised when a scope has no headroom left for the estimated cost."""

class BudgetGuard:
    def __init__(self, client: redis.Redis, limits: dict):
        self.client = client
        self.limits = limits          # {(scope, period): Decimal}
        self.reserve_script = client.register_script(RESERVE_LUA)

    @staticmethod
    def estimate_usd(prompt: str, max_output_tokens: int,
                     price_in_per_m: Decimal, price_out_per_m: Decimal) -> Decimal:
        # Heuristic tokeniser: ~4 chars/token for Latin, ~1.7 for CJK,
        # plus 24 tokens of chat framing. Reconcile afterwards, always.
        ascii_chars = sum(1 for ch in prompt if ord(ch) < 128)
        wide_chars = len(prompt) - ascii_chars
        input_tokens = ascii_chars // 4 + int(wide_chars / 1.7) + 24
        return (input_tokens * price_in_per_m
                + max_output_tokens * price_out_per_m) / Decimal(1_000_000)

    def reserve(self, scope: str, period: str, amount_usd: Decimal,
                seconds_left: int) -> Reservation:
        limit = self.limits[(scope, period)]
        key = "budget:{}:{}".format(period, scope)
        total, remaining = self.reserve_script(
            keys=[key],
            args=[str(amount_usd), str(limit), int(seconds_left)],
        )
        if float(total) == -1:
            raise BudgetExceeded(
                "{} / {} has {:.4f} USD headroom left".format(scope, period, float(remaining))
            )
        return Reservation(uuid.uuid4().hex, scope, period, amount_usd)

    def reconcile(self, reservation: Reservation, actual_usd: Decimal) -> None:
        # Refund unused headroom, or charge the overrun. Redis preserves the TTL.
        key = "budget:{}:{}".format(reservation.period, reservation.scope)
        delta = actual_usd - reservation.amount_usd
        if delta != 0:
            self.client.incrbyfloat(key, str(delta))

# Call site: reserve against every scope that applies, innermost first.
guard = BudgetGuard(redis.Redis(), limits={
    ("org", "month"): Decimal("5000"),
    ("user:u_8812", "day"): Decimal("25"),
})

estimate = guard.estimate_usd(prompt, 800, Decimal("3.00"), Decimal("15.00"))
reservation = guard.reserve("user:u_8812", "day", estimate, seconds_left=43200)
try:
    response = call_model(prompt, max_output_tokens=800)
finally:
    # On failure actual usage is zero, so the reservation is refunded in full.
    guard.reconcile(reservation, Decimal(str(response.usage.cost_usd)))

The ledger is the durable half: the counter in front of it is a cache, the ledger is what you reconcile against the provider invoice. Note the FOR UPDATE on the limit row, which serialises concurrent reservations for the same scope and makes the conditional insert safe under load.

CREATE TABLE budget_reservations (
  request_id    uuid          PRIMARY KEY,
  trace_id      uuid          NOT NULL,
  tenant_id     text          NOT NULL,
  feature       text          NOT NULL,
  model         text          NOT NULL,
  scope         text          NOT NULL,
  scope_id      text          NOT NULL,
  period        text          NOT NULL,
  period_start  timestamptz   NOT NULL,
  reserved_usd  numeric(14,6) NOT NULL,
  settled_usd   numeric(14,6),
  state         text          NOT NULL DEFAULT 'reserved'
                  CHECK (state IN ('reserved', 'settled', 'released')),
  created_at    timestamptz   NOT NULL DEFAULT now(),
  settled_at    timestamptz
);

CREATE INDEX budget_reservations_scope_idx
  ON budget_reservations (scope, scope_id, period, period_start);
CREATE INDEX budget_reservations_trace_idx  ON budget_reservations (trace_id);
CREATE INDEX budget_reservations_tenant_idx ON budget_reservations (tenant_id, created_at DESC);

-- Reserve. The row lock serialises concurrent reservations for this scope,
-- so two requests cannot both observe the same headroom.
BEGIN;

SELECT limit_usd FROM budget_limits
WHERE scope = :scope AND scope_id = :scope_id AND period = :period
FOR UPDATE;

INSERT INTO budget_reservations (
  request_id, trace_id, tenant_id, feature, model,
  scope, scope_id, period, period_start, reserved_usd
)
SELECT :request_id, :trace_id, :tenant_id, :feature, :model,
       :scope, :scope_id, :period, :period_start, :estimate_usd
FROM (
  SELECT :limit_usd - COALESCE(sum(COALESCE(settled_usd, reserved_usd)), 0) AS headroom
  FROM budget_reservations
  WHERE scope = :scope AND scope_id = :scope_id
    AND period = :period AND period_start = :period_start
    AND state <> 'released'
) AS h
WHERE h.headroom >= :estimate_usd
RETURNING request_id;

COMMIT;
-- Zero rows returned means the budget was exhausted: reject with HTTP 402 or 429.

-- Settle with real usage from the provider response.
UPDATE budget_reservations
SET state = 'settled', settled_usd = :actual_usd, settled_at = now()
WHERE request_id = :request_id AND state = 'reserved';

-- Attribution: what is spending, right now, for this tenant.
SELECT feature, model,
       count(*)                                   AS calls,
       sum(COALESCE(settled_usd, reserved_usd))   AS usd
FROM budget_reservations
WHERE tenant_id = :tenant_id
  AND created_at >= now() - interval '15 minutes'
GROUP BY feature, model
ORDER BY usd DESC
LIMIT 10;

The minimum viable control set for week one

With one week, I would ship five things and deliberately skip the rest.

1. Tag every request with tenant, user, feature and trace id. Nothing else works without this.
2. Reserve and reconcile against an atomic counter at the org and per-user scopes. Two scopes, one script, one call site.
3. Set the 70/90/100 ladder with a real degradation path at 90 percent. If the 90 percent tier does nothing, you have not shipped a ladder.
4. Ship one global and one per-tenant kill switch, read from a cached flag, with a documented flip procedure and a shadow-mode test you have actually run.
5. Add three alerts: spend rate versus trailing baseline, tokens per request z-score, and requests per user per minute — minute-level evaluation, top spenders in the alert body.

Defer per-feature budgets until the taxonomy stabilises, the 120 percent tier until you trust reservations, and ML-based anomaly detection entirely: rules catch the failure modes above, and an untrusted alert is an ignored alert.

Frequently asked questions

Should the budget be enforced in the application or at the gateway?

Both, with different jobs. The application owns the semantic caps only it can see — per-user daily limits, per-feature budgets, maximum turns per agent run — and passes a pre-flight estimate downstream. The gateway owns the ceilings that must hold regardless of which code path is calling. Application-only enforcement fails the moment someone adds a service that calls the provider directly, and it fails silently. If you ship one first, ship the gateway: it is the only layer that cannot be bypassed.

How accurate does pre-flight estimation need to be?

Accurate enough that the reservation is a useful bound. A character-based heuristic lands within roughly 15 percent for typical chat traffic, and reservation over-reserves anyway because it charges max_tokens rather than expected output. Reconciliation keeps the system honest: reserved amounts are provisional, settled amounts are truth, and the counter is corrected by the delta. Estimates consistently off by more than 30 percent point at your callers’ max_tokens settings, not at the guard.

Why not just set a hard cap and be done with it?

Because the first thing users experience is total failure with no warning, usually at an inconvenient hour. A hard cap with no ladder also fails badly at the org level: one runaway job exhausts the shared budget and every other tenant is blocked, turning a single bad deploy into a platform-wide outage. The ladder exists so the system changes behaviour before it stops responding.

How do I know my kill switch will actually work?

By using it in production when nothing is wrong. Flip it for an internal tenant or one percent of traffic on a schedule, verify callers receive the expected error code and that retries do not amplify the rejection into a storm, then time the interval from flip to effect and treat it as an SLO. A kill switch never exercised under production load is an untested branch in the most important code path you own.

Conclusion

Runaway AI spend is a bounded problem with a small set of known causes. The failure modes are structural — loops without termination, retries without budgets, bulk actions without per-user caps, jobs without ceilings — and each has a signature that appears in aggregates long before it appears on an invoice. What makes guardrails work is coverage, not sophistication: tag everything, reserve atomically before dispatching, reconcile after, enforce at the narrowest scope that covers the blast radius, and keep a gateway as the line that cannot be bypassed.

Related reading

• How to Reduce AI API Costs: A Practical Guide for Developers
• Metering and Billing AI Usage Per User: A Practical SaaS Guide
• Prompt Caching Explained: How to Cut Costs on Repeated Context
• Semantic Caching for AI APIs: Cut Latency and Cost by Up to 60%

An LLM API bill is priced per token, not per request and not per word: two rates for input and output, plus a third cheaper rate for input the provider has already seen. Caching, batching, model choice and reasoning effort are all ways of moving tokens between those three buckets.

Every rate below is illustrative, chosen so the arithmetic stays visible. Real rates differ between providers and change.

The billing unit is the token

A token is a subword unit produced by the model’s tokenizer. For clean English prose, four characters per token is the usual rule of thumb, which puts roughly 750 words at roughly 1,000 tokens — fine as a sanity check, dangerous as a budget input.

Structured content tokenises worse. Code lands nearer three characters per token. Base64 blobs, hex digests, UUIDs and long digit strings are close to the worst case, because the tokenizer has no vocabulary for them and falls back to fragments. CJK text is a different regime: one Chinese character is often one token, so a document a quarter the length of an English one costs the same.

Content	Approximate characters per token	What drives the cost
English prose	4	Common subwords are single tokens
Source code	3	Short identifiers, punctuation, indentation
JSON and YAML	2.5 to 3	Every key name, quote, brace and comma is billed
Base64, hex, UUIDs	1.5 to 2	No vocabulary; tokenizer emits fragments
CJK text	1 to 1.5	Whole characters are single tokens
Long digit strings	1.5 to 2	Digit grouping is inconsistent across tokenizers

JSON pays for schema, not information: twenty keys averaging ten characters spend roughly 250 characters, on the order of 90 tokens, on key names, quotes and colons before a single value is transmitted. An estimator that splits on whitespace sees a minified JSON blob as one word and is wrong by an order of magnitude.

Input tokens versus output tokens

Input is everything the model reads; output is everything it writes, and output typically costs several times more per token.

The reason is mechanical. Prompt processing is a single forward pass over a known sequence, so the work parallelises across the accelerator’s compute units. Generation is sequential by construction: token n+1 cannot be computed until token n exists, so the model runs one step at a time, each step reading the entire growing key-value state. That workload is memory-bandwidth-bound rather than compute-bound, so cost per token is much higher.

Illustrative tier	Input USD / million	Cached input USD / million	Cache write USD / million	Output USD / million	Batch output USD / million
Small and fast	0.25	0.025	0.31	1.25	0.625
Mid-tier general	3.00	0.30	3.75	15.00	7.50
Frontier reasoning	15.00	1.50	18.75	75.00	37.50

Output is the expensive bucket per token; input is the large bucket by volume. In the worked example below, input outnumbers output roughly 18 to 1 in tokens, turning a fivefold price ratio into 3.6-fold in dollars. Tuning the completion while ignoring the prompt optimises the smaller half of the bill.

The complete token bill for one request

All of the following is input, and all of it is re-billed on every call:

• The system prompt: persona, policy and output-format rules.
• Tool schemas: descriptions, parameter names, enums.
• Retrieved context: RAG chunks, search results, file excerpts.
• The full conversation history so far.
• The current user message.

Output is the visible completion plus, on reasoning models, the internal tokens generated before the answer begins.

Multi-turn chat grows quadratically

A stateless completion API keeps no session state, so a chat client re-sends the entire history every turn. Cost per turn grows linearly with turn number, so total cost grows with the square of conversation length. For a ten-turn conversation with a 2,500-token system and tool block, 150-token user messages and 350-token replies:

• Input on turn 1: 2,500 + 150 = 2,650 tokens.
• Input on turn 10: 2,500 + (9 x 500) + 150 = 7,150 tokens.
• Total input: (10 x 2,650) + 500 x (0 + 1 + … + 9) = 26,500 + 22,500 = 49,000 tokens.
• Output: 10 x 350 = 3,500 tokens.
• Unique content is only 2,500 + 1,500 + 3,500 = 7,500 tokens, so you paid 6.5 times over for the privilege of being stateless.

At the illustrative mid-tier rates that is 49,000 x 3.00 / 1,000,000 = 0.147 USD of input plus 3,500 x 15.00 / 1,000,000 = 0.0525 USD of output, about 0.20 USD. Twenty turns cost 148,000 input tokens — three times the total for twice the turns.

Cached input: what a prompt cache actually caches

A prompt cache stores the model’s internal key-value state for a prefix of the prompt. If a later request has a byte-identical prefix in the same position, the provider skips the prefill computation for those tokens and bills them at the cached-input rate: typically 80 to 90 percent below fresh input, plus a small surcharge on the write that populates the cache. Two conditions decide whether it fires, both of them design decisions.

1. The prefix must be byte-identical. One character different — a timestamp, a user name, a reordered tool list — and every request misses. Serialising JSON dictionaries without sorting keys is enough to break it.
2. The shared prefix must come first. Caches match forward from the start of the prompt, so anything variable placed before the static block invalidates everything after it.

The ordering mistake that quietly costs a quarter of the bill

The natural way to write a template is to put the per-user instruction where a human would read it, at the top. So teams build f"You are assisting {user_name} at {company}. " + SHARED_SYSTEM_PROMPT and then wonder why the cache never hits. Move every variable — user name, tenant, locale, current date, retrieved documents — after the static system prompt and tool schemas. The template becomes less readable and the bill drops.

Retrieved context is variable by definition and can never be in the cached prefix, so order multiple retrievals most-stable-first. TTL is the other lever: caches expire after a few minutes of inactivity, and some providers sell a longer window at a higher write cost. Continuous traffic keeps a short TTL warm for almost no write overhead; bursty traffic means paying the write repeatedly for prefixes nobody reuses. Treat the prefix as a versioned interface with a monitored hit ratio: a template change that drops that ratio from 0.95 to zero is a large cost increase no deploy diff will show you. The prompt caching guide covers prefix construction.

Reasoning tokens are output tokens

Models that reason before answering generate internal tokens billed at the output rate even though they never appear in the returned text. Some APIs report them separately as reasoning_tokens, others fold them into the completion count, and a few never surface them.

Output length therefore stops being a property of your prompt and becomes a property of the problem. A request producing a 400-token answer can generate 4,000 reasoning tokens on a hard input: at the illustrative mid-tier rate, 4,000 x 15.00 / 1,000,000 = 0.06 USD of invisible output against 0.006 USD of visible output.

This is how a cheap model becomes expensive. A small model at 1.25 USD per million output tokens is genuinely a tenth of the price per token. But if it fails to converge and burns 30,000 reasoning tokens, that call costs 0.0375 USD, while a stronger model settling in 2,000 tokens costs 0.03 USD. The cheap model lost on price and produced the worse answer. Cap the effort or thinking-budget setting where a provider offers one, log reasoning tokens per request, and alert on the p95 rather than the mean. The reasoning models guide covers picking an effort level per task class.

Batch APIs trade latency for price

Asynchronous batch endpoints accept a file of requests and return results within a turnaround window, commonly up to 24 hours, in exchange for a substantial discount on both input and output — often around half. If nothing user-facing is waiting, batch is free money; if something is waiting, it is not a discount at all. Nightly summarisation, embedding backfills, eval scoring, bulk classification and moderation sweeps qualify; anything inside a request path does not.

Three failure modes to design around. Results expire, so a job read a day after it completes may return nothing. Failures arrive per request rather than as a job-level error, so you need per-line reconciliation and a resubmission path. And a job that fails validation on submission still consumed the upload, so validate the JSONL locally first. The batch processing guide walks through the job lifecycle.

Non-text modalities are metered on different units

Images are priced per image with the price derived from resolution — by tiling or by normalising to a pixel budget — so a 4K screenshot can cost more than several pages of text. Audio input is priced per second of audio and silence is billed; audio output is priced per character or per second. Video is priced per second or by sampled frames billed at the per-image rate, so a 60-second clip can exceed the entire text budget around it.

Modality	Metering unit	Relative magnitude	What surprises people
Image input	Per image, derived from resolution or tiles	Comparable to a long text prompt	Downscaling often costs no accuracy and cuts this substantially
Image generation	Per image, by size and quality	Orders above text generation	Size and quality settings multiply, they do not add
Audio input	Per second of audio	Cheap per minute	Silence and hold music are billed
Audio output	Per character or per second	Cheap per thousand characters	Long-form narration dominates the request
Video input	Per second, or per sampled frame	Usually the most expensive unit here	Frames multiply the per-image rate by frame count
Embeddings	Per input token	An order of magnitude below generation	Every re-index pays again for the whole corpus

Log the modality breakdown separately. A single multimodal request can dominate a monthly bill while looking like one request in your metrics.

The cost multipliers nobody models

• Retries. A retry bills twice, and if the first attempt failed after prompt processing you paid input for output you never received. A 3 percent retry rate adds 3 percent; a retry that resends an enlarged context adds far more.
• Guardrail-triggered regeneration. If 4 percent of completions fail validation and are regenerated, output cost rises 4 percent — plus the validator, which for an LLM-as-judge check is a second inference per request.
• System prompt duplication. A 2,500-token system prompt on 600,000 requests a month is 1,500 million input tokens, or 4,500 USD at the illustrative mid-tier rate. It is often the largest single line item and it is invisible in any dashboard reporting per-request cost.
• Verbose tool schemas. Twenty tools at 250 tokens each is 5,000 tokens per call, paid on every turn including turns that invoke no tool.
• Side costs. Embedding every chunk for a vector store, re-embedding on each re-index, and observability platforms billing per span or stored event.

Worked example: a support chatbot

Assume a static system prompt of 1,800 tokens, static tool schemas of 700, retrieved context of 2,500, average history of 1,200 and a user message of 150. Input is 1,800 + 700 + 2,500 + 1,200 + 150 = 6,350 tokens. Output is 350 tokens with no reasoning. Volume is 20,000 requests per day, 600,000 per month. Rates: 3.00 USD per million input, 15.00 per million output, 0.30 per million cached input, 3.75 per million cache write.

Baseline. Input: 600,000 x 6,350 = 3,810 million tokens at 3.00 USD = 11,430 USD. Output: 600,000 x 350 = 210 million tokens at 15.00 USD = 3,150 USD. Total 14,580 USD per month.

With prompt caching. The cacheable prefix is the system prompt plus tool schemas: 2,500 tokens, byte-identical, positioned first. Assume a five-minute TTL and continuous traffic, so essentially every request hits. Cached tokens: 600,000 x 2,500 = 1,500 million at 0.30 USD = 450 USD, against 4,500 USD uncached — a saving of 4,050 USD. Cache writes: 288 TTL windows per day x 2,500 tokens = 21.6 million tokens per month at 3.75 USD = 81 USD. New total: 11,430 – 4,050 + 81 + 3,150 = 10,611 USD, a 27 percent reduction. At a 0.90 hit ratio, add back 150 million tokens at 3.00 USD = 450 USD, for about 11,061 USD.

Adding batch for the offline slice. Suppose 30 percent of volume is a nightly transcript-scoring job with no user waiting: 180,000 requests per month, routed to the batch endpoint at a 50 percent discount on uncached input and output. Those requests still read the cached prefix at 135 USD. Their uncached input is 180,000 x 3,850 = 693 million tokens at 1.50 USD = 1,039.50 USD, and their output is 180,000 x 350 = 63 million tokens at 7.50 USD = 472.50 USD, for a slice total of 1,647 USD against 3,159 USD run synchronously. The remaining 420,000 online requests cost 315 + 4,851 + 2,205 = 7,371 USD. Grand total: 9,099 USD per month, 37.6 percent below baseline.

Scenario	Uncached input USD	Cached input USD	Cache writes USD	Output USD	Monthly total USD
Baseline	11,430.00	0.00	0.00	3,150.00	14,580.00
Prompt caching on the static prefix	6,930.00	450.00	81.00	3,150.00	10,611.00
Caching plus batch for 30 percent of volume	5,890.50	450.00	81.00	2,677.50	9,099.00

Forecasting before you launch

The formula: requests per day times 30, times the sum of uncached input tokens over a million times the input rate, cached prefix tokens over a million times the cached rate, and output tokens over a million times the output rate, plus side costs.

Two numbers dominate the uncertainty and neither is the price. The first is tokens per request, which in a chat product is not a constant: it grows with conversation length, retrieval depth and tool-catalogue size, and a p50-to-p95 spread of three times is normal. The second is requests per day, easy to underestimate for anything with fan-out — an agent that loops, or a nightly job that scales with the size of a customer’s data rather than the number of customers.

Price changes are the third-order risk: public, announced, and applying to everyone. Instrument real token counts from the first request, split by feature and tenant, and reconcile them against the invoice monthly — the gap between what your logs say and what you were billed for is where retries, failed attempts and side costs hide. The usage metering and billing pipeline makes that reconciliation possible, and LLM observability is where per-request counts should already be flowing. A gateway reporting usage across providers behind one key gives you the minimum required, which is the role qoraapi.com plays.

Counting tokens and projecting cost in Python

The first script counts real tokens and compares a JSON serialisation against the same facts as prose.

import json
import tiktoken

ENC = tiktoken.get_encoding("o200k_base")

def n_tokens(text: str) -> int:
    return len(ENC.encode(text, disallowed_special=()))

system_prompt = open("prompts/support_v7.md", encoding="utf-8").read()
tool_schemas = json.dumps(json.load(open("tools/support_tools.json", encoding="utf-8")))

def input_tokens(history, retrieved_docs, user_message):
    """Everything the model reads is input. Miss one part and the forecast is wrong."""
    parts = [system_prompt, tool_schemas]
    parts += [turn["content"] for turn in history]
    parts += [doc["text"] for doc in retrieved_docs]
    parts.append(user_message)
    return sum(n_tokens(p) for p in parts)

record = {
    "invoice_id": "INV-2026-00418",
    "customer": {"id": "a3f9b2c1-77d4-4e8a-9b12-5c6d7e8f9a0b", "plan": "growth", "seats": 42},
    "line_items": [
        {"sku": "SKU-99183-A", "qty": 2, "unit_price_cents": 74950},
        {"sku": "SKU-10277-C", "qty": 1, "unit_price_cents": 12900},
    ],
    "subtotal_cents": 162800,
    "tax_cents": 14652,
    "total_cents": 177452,
    "currency": "USD",
    "due_date": "2026-04-14",
    "status": "open",
}

as_json = json.dumps(record, separators=(",", ":"))
as_prose = (
    "Invoice INV-2026-00418 for customer a3f9b2c1-77d4-4e8a-9b12-5c6d7e8f9a0b "
    "on the growth plan with 42 seats is open, due 2026-04-14. Two units of "
    "SKU-99183-A at 74950 cents and one unit of SKU-10277-C at 12900 cents. "
    "Subtotal 162800, tax 14652, total 177452 USD."
)

print(f"json   chars={len(as_json):>4}  tokens={n_tokens(as_json):>4}")
print(f"prose  chars={len(as_prose):>4}  tokens={n_tokens(as_prose):>4}")
print(f"json costs {n_tokens(as_json) / n_tokens(as_prose):.2f}x the prose")

Expect that ratio to land between 1.4 and 2 times, and to drift as your key names get longer.

The second script projects a monthly bill, including a cache-hit assumption and a batch share.

from dataclasses import dataclass

RATES = {
    "small":    {"in": 0.25, "cached": 0.025, "out": 1.25},
    "mid":      {"in": 3.00, "cached": 0.30,  "out": 15.00},
    "frontier": {"in": 15.00, "cached": 1.50, "out": 75.00},
}
CACHE_WRITE_MULTIPLIER = 1.25   # a cache write usually costs slightly more than fresh input
BATCH_DISCOUNT = 0.50           # applied to uncached input and to output
CACHE_WRITES_PER_DAY = 288      # five-minute TTL, steady traffic

@dataclass
class Workload:
    requests_per_day: int
    cacheable_prefix_tokens: int   # must be byte-identical AND first in the prompt
    dynamic_input_tokens: int      # history + retrieval + user message
    output_tokens: int
    cache_hit_ratio: float = 0.0
    batch_share: float = 0.0

def monthly_cost(w: Workload, tier: str = "mid", days: int = 30) -> float:
    r = RATES[tier]
    reqs = w.requests_per_day * days
    batch_reqs = reqs * w.batch_share

    def slice_cost(n: float, discount: float) -> float:
        cached = n * w.cacheable_prefix_tokens / 1e6 * w.cache_hit_ratio
        uncached_prefix = n * w.cacheable_prefix_tokens / 1e6 * (1 - w.cache_hit_ratio)
        dynamic = n * w.dynamic_input_tokens / 1e6
        output = n * w.output_tokens / 1e6
        return (
            cached * r["cached"]                       # cache reads are already discounted
            + (uncached_prefix + dynamic) * r["in"] * discount
            + output * r["out"] * discount
        )

    writes = CACHE_WRITES_PER_DAY * days * w.cacheable_prefix_tokens / 1e6
    write_cost = writes * r["in"] * CACHE_WRITE_MULTIPLIER * (w.cache_hit_ratio > 0)
    return slice_cost(reqs - batch_reqs, 1.0) + slice_cost(batch_reqs, BATCH_DISCOUNT) + write_cost

base = dict(requests_per_day=20_000, cacheable_prefix_tokens=2_500,
            dynamic_input_tokens=3_850, output_tokens=350)

scenarios = [
    ("baseline, no cache",   Workload(**base)),
    ("cached prefix",        Workload(**base, cache_hit_ratio=1.0)),
    ("cached + 30% batch",   Workload(**base, cache_hit_ratio=1.0, batch_share=0.30)),
    ("cached, 0.9 hit rate", Workload(**base, cache_hit_ratio=0.90)),
]

for label, workload in scenarios:
    print(f"{label:<22} {monthly_cost(workload):>10,.0f} USD / month")

That prints roughly 14,580, 10,611, 9,099 and 11,061 USD — the same four numbers derived by hand above. Change cache_hit_ratio first; it is the input most likely to be wrong.

Frequently asked questions

Do I pay for a request that fails?

Usually yes, at least partially: input tokens are billed once the prompt has been processed, and a provider returning a 500 after prefill has already done that work. Fail fast with a short timeout so you do not pay for a prefill that will be discarded, and do not retry requests that failed on validation rather than infrastructure.

Why did my bill double when my request count stayed flat?

Because tokens per request is not a constant. Usual causes, in rough order of frequency: history growing because nothing is evicted; retrieval returning more chunks after an index change; a tool catalogue growing from eight entries to thirty; a model upgrade that emits reasoning tokens where the previous version did not; and a template edit that broke the cache prefix. All five show up in a per-request token histogram and never in a request-count graph.

Is prompt caching the same as semantic caching?

No, and the confusion costs money. A prompt cache reuses key-value state for a byte-identical prefix inside the provider’s inference stack, and cannot return a wrong answer because it changes nothing about the computation. A semantic cache matches a new request against previously answered similar requests in your application, and can return a stale answer, so it needs a similarity threshold and an invalidation strategy. A semantic cache hit removes the request entirely; a prompt cache hit only discounts part of it.

Can I forecast a bill without running the workload?

Within a factor of two or three, which is not enough to price a product on. The uncertainty is not in the published rates but in tokens per request and requests per day, both of which depend on behaviour you have not observed yet. Build the instrumented version first and run it on real traffic for a week. If you need a number immediately, forecast the p95 rather than the mean.

Conclusion

The pricing model is three rates — fresh input, cached input, output — applied to counts you control. The arithmetic is not complicated, which is exactly why the surprises are embarrassing: they come from not counting something, not from mispricing it.

The single pricing decision that most often surprises teams is prompt layout, and it is made in week one by whoever writes the template. Putting a per-user instruction before a shared 2,000-token system prompt is reasonable for readability and it forfeits a discount of 80 to 90 percent on the largest token category in the request. Nothing breaks: the model still answers, and the invoice is simply larger than it needed to be for the life of the product.

So take the position explicitly. Order the prompt static-first and variable-last, treat the prefix as a versioned interface with a monitored hit ratio, cap reasoning effort per task class, route anything offline to the batch endpoint, and instrument real token counts from day one. Those five things turn a bill you discover into a number you predicted.

Related reading

• How to Reduce AI API Costs: A Practical Guide for Developers
• Metering and Billing AI Usage Per User: A Practical SaaS Guide
• Prompt Caching Explained: How to Cut Costs on Repeated Context
• Batch AI APIs: Processing Millions of Requests Affordably

An A/B test is the only mechanism that reliably tells you whether a prompt edit or a model swap made your product better, and most teams run them badly enough that the result is worse than no test at all. If your feature calls an LLM, the prompt is production code with no type checker, no unit test covering real inputs, and a behaviour that shifts when a provider updates a model behind the same alias. The experiment is your regression suite.

Why an LLM change needs an experiment at all

A prompt edit is a code change with no compiler. Change “Summarise the ticket” to “Summarise the ticket in one paragraph, no bullet points” and every unit test still passes, because tests assert on JSON shape and field presence, not on whether the summary still preserves the escalation reason. Nothing in CI can see that regression.

Offline evals do not predict production

Eval sets are written by the people who wrote the prompt, from the cases they had in mind. Production traffic is long-tailed: pasted stack traces, mixed-language input, empty strings, 40,000-token documents, users typing “no” into a date field. A 200-example eval set proves you did not break the cases you already considered, and says nothing about the strange 3% where the support tickets come from. Run the offline eval harness as a cheap filter, then experiment on the survivors.

Provider-side changes are unannounced

Model aliases are pointers. A dated snapshot is safer than a floating alias, but even pinned snapshots get silent infrastructure and quantisation changes. Log the version returned in the response, not the alias you requested, or you will attribute a provider-side change to your own prompt edit.

What is actually random here

Three sources of variance, and conflating them is the most common design error. Model non-determinism covers sampling above temperature zero plus batch-size-dependent kernels and mixture-of-experts routing: temperature 0 removes sampling noise but not byte-identical output across calls, let alone across versions. Traffic mix is who arrives during the window, and in most real experiments it dominates. Assignment maps a unit to a variant, and becomes a noise source rather than a control if it is not deterministic and stable.

So never let the model be both the treatment and the source of noise. If arms differ in temperature as well as in prompt, the measured difference mixes the prompt effect with sampling variance, and one request per condition cannot separate them. Fix temperature, top_p and seed across arms; vary exactly one thing.

Unit of randomisation: user, session, or request

The unit decides whether you measure a user-visible effect or a statistical artefact.

Unit	Assignment key	Contamination risk	Statistical power	Use when
Request	request_id	High: one user sees both behaviours inside a session	Highest, because units are plentiful	Internal batch jobs nobody reads, genuinely independent requests
Session	session_id	Medium: consistent within a conversation, not across them	Medium	A session is the entire unit of value and there is no account
User or tenant	user_id, tenant_id	None	Lowest, because correlated outcomes shrink effective n	Anything a human sees, anything with memory, anything touching retention

Per-request assignment contaminates the experience: the same user watches the assistant answer in one style and then another, with different refusals and different latency. Worse, the user’s behaviour then depends on the mix of variants they received, which breaks the comparison you wanted.

Per-session assignment looks like a middle ground and is mostly a trap. Sessions are short, correlated and interleaved for the same user, so you still get visible inconsistency and still have to cluster variance at the user level. If you are clustering at the user level anyway, you had user-level power all along.

Recommendation: randomise by user, or by tenant for B2B products. The power cost is real, and the section below quantifies it.

Sticky assignment: hash, do not store

Assignment should be a pure function of experiment id, unit id and salt. No storage, no cookie, no database read on the hot path.

import hashlib
import struct

def assign_variant(experiment_id: str, unit_id: str,
                   weights: dict, salt: str = "v1") -> str:
    """Deterministic variant assignment.

    The same (experiment_id, unit_id, salt) maps to the same variant on every
    service, in every language, forever. No state to read, no state to lose.
    """
    total = sum(weights.values())
    key = "{0}:{1}:{2}".format(experiment_id, unit_id, salt).encode("utf-8")

    # First 8 bytes of SHA-256 as an unsigned big-endian integer, bucketed
    # into [0, total). SHA-256 is used as a hash here, not for security.
    bucket = struct.unpack(">Q", hashlib.sha256(key).digest()[:8])[0] % total

    cursor = 0
    for variant, weight in weights.items():
        cursor += weight
        if bucket < cursor:
            return variant

    raise AssertionError("unreachable: bucket out of range")

# 90 / 10 canary split. Changing the weights REQUIRES a new salt, otherwise
# users already measured get silently reassigned and both arms are corrupted.
weights = {"control": 90, "candidate": 10}
variant = assign_variant("checkout-summary-v3", user_id, weights, salt="v1")

Why not a cookie alone: cookies are mutable, cleared, per-device and increasingly blocked. A user who clears cookies gets re-randomised, and those users are not a random sample, so you have injected selection bias into your assignment. Persist it server-side for audit, but derive it from the hash.

Use a different salt per experiment, or the same users land in bucket zero for every test and your portfolio-wide control group becomes a fixed, non-random subset. Freeze the salt at launch: re-weighting a running experiment silently reassigns users you have already measured.

Metrics: one primary, everything else is a guardrail

Class	Metric	Definition	Failure it catches
Primary	Task success	Thumbs up, ticket resolved, suggestion accepted, downstream conversion	Whether the change helped at all
Guardrail	p95 latency	End-to-end including retries	Perceived slowness, client timeouts
Guardrail	Cost per successful task	Total spend divided by successes	A cheap model that retries its way back to expensive
Guardrail	Refusal and empty-response rate	Responses that decline or return nothing usable	Over-aligned prompt, truncated context
Guardrail	Error and retry rate	5xx, timeouts, schema validation failures	Provider breakage, parser drift
Guardrail	Safety incidents	Policy violations, PII leakage, unsafe tool calls	Compliance exposure
Diagnostic	Tokens, cache hit rate, tool calls	Not decision metrics	Explaining a move, not deciding one

There should be exactly one primary metric; three means none, because you will report whichever moved.

Cost and latency are guardrails, not tie-breakers. Pre-register non-inferiority bounds: the candidate may not raise p95 latency by more than 15%, cost per successful task by more than 5%, or the refusal rate by more than two percentage points. A breach is a loss regardless of the primary metric, because a quality win that triples the inference bill is a product decision rather than a test result.

Emit diagnostics on the same event stream as the primary metric, so attribution does not require joining three systems later. That is what makes LLM observability useful rather than decorative.

Measuring quality when quality is not a number

Pairwise preference beats absolute scoring

Ask raters to pick between two outputs for the same input rather than score one from 1 to 5, because absolute scales drift between raters and within a rater across a session. Randomise left and right order to control position bias, blind the rater to variant identity, and measure agreement first: if two humans agree 55% of the time, the metric is noise. An LLM judge needs validating against human labels on a held-out sample, because a judge from the candidate’s own model family prefers its own outputs.

Implicit signals, and the proxy trap

Every request produces traces without asking: copy-to-clipboard, acceptance of a suggestion, edit distance between suggested and sent text, regeneration, abandonment, escalation to a human. They arrive at request scale, so they reach significance far sooner than a thumbs-up that 2% of users click. They are also gameable: optimising copy rate rewards long outputs, optimising edit distance rewards outputs that resemble the user’s draft, optimising retry rate rewards a model that is confidently wrong. Anchor at least one metric to something the business already counts, such as revenue or time-to-close: a dense proxy plus a sparse real outcome is workable, a dense proxy alone is a metric you will optimise into absurdity.

Sample size: the arithmetic that kills most experiments

For a two-arm test on a proportion, the sample per arm is n = (z_alpha + z_beta)^2 * (p1(1-p1) + p2(1-p2)) / (p1-p2)^2. At alpha 0.05 two-sided and 80% power the z values are 1.96 and 0.84, so the squared sum is 7.84. Take a baseline task-success rate of 12%.

• Detect a 10% relative lift, 12% to 13.2%: p1(1-p1) = 0.1056, p2(1-p2) = 0.1146, sum 0.2202, and (p1-p2)^2 = 0.000144. n = 7.84 x 0.2202 / 0.000144 = 11,990 per arm.
• Detect a 5% relative lift, 12% to 12.6%: sum 0.2157, (p1-p2)^2 = 0.000036, so n = 46,970 per arm.
• Detect a 2% relative lift, 12% to 12.24%: sum 0.2130, (p1-p2)^2 = 0.00000576, so n = 289,900 per arm.

Those are independent observations, and users are not independent of each other. At 20 requests per user and an intra-user correlation of 0.3, the design effect is 1 + 19 x 0.3 = 6.7. Reaching the equivalent of 11,990 independent observations needs roughly 11,990 x 6.7 = 80,000 requests per arm, or about 4,000 users. The same multiplier on the 2% lift means 1.94 million requests per arm, 3.9 million across both arms: at 50,000 requests a day, 78 days.

You cannot detect a 2% relative quality win at that traffic. Not “it is hard”: you cannot, and a test that reports significance there reports noise. Decide the minimum detectable effect before launch, and if the arithmetic says 78 days, do not run the test. Ship behind a canary with guardrails, reduce the noise in the metric, or accept the change on qualitative grounds and say so out loud. Extra arms make this worse: each one needs its own full sample, and with k arms the family-wise error rate at alpha 0.05 is 1 – 0.95^(k-1), which is 14% at four arms.

Peeking, novelty, and the calendar

The peeking problem

Stop the first time p drops below 0.05 and your false-positive rate is not 5%; it is much higher, approaching 1 with enough looks. Under the null your test statistic is a random walk, and the chance it crosses a fixed boundary before your planned sample size far exceeds nominal alpha. A fixed-horizon test guarantees 5% only if you look once, at the pre-registered n.

Two legitimate fixes. A fixed horizon: freeze the sample size and analysis, look once, decide. If you need interim monitoring, use group-sequential boundaries with alpha spending, where O’Brien-Fleming is standard for a small number of looks. Or design for continuous monitoring from the start, using always-valid confidence sequences or a Bayesian rule with a pre-registered posterior threshold. What is not legitimate is a stopping rule chosen after seeing the data. Guardrail breaches are the exception: stopping for harm is safety, not efficacy.

Novelty, primacy, and seasonal traffic

Users engage more with anything new and habituate to anything that persists, and both effects decay over days. A two-day test measures the novelty spike rather than the steady state you are shipping. Run at least one full week, and prefer two when the metric is user-mediated; mechanical metrics such as latency and cost need no such window.

Do not run an experiment across a holiday, a marketing campaign, a pricing change or a product launch. Traffic mix shifts and the control group stops being a valid counterfactual. If a launch is unavoidable, log it as a covariate and plan to re-run.

Model swaps: cost per successful task, not cost per call

Worked example, 10,000 requests per arm. Incumbent model A costs $0.0040 per call and succeeds 92% of the time with no retries. Candidate B costs $0.0012 per call and succeeds 84% on the first attempt; your client retries 25% of first-attempt failures once, and a retry succeeds 84% of the time at the same price.

• A: cost is 10,000 x $0.0040 = $40.00, successes are 9,200, so cost per success is $40.00 / 9,200 = $0.00435.
• B: first-attempt failures are 1,600, of which 400 are retried. Retry cost is 400 x $0.0012 = $0.48, total cost $12.00 + $0.48 = $12.48. Retry successes are 400 x 0.84 = 336, total successes 8,400 + 336 = 8,736, so cost per success is $12.48 / 8,736 = $0.00143.

B is about three times cheaper per success, so on cost alone it wins. Now value the success: at $0.50 of downstream value per task, A generates 9,200 x $0.50 = $4,600 and B generates 8,736 x $0.50 = $4,368. B saves $27.52 in inference and gives up $232.00 in value. Cost per call hid that.

A slower model can push your p95 past a client timeout, and the resulting retries and abandonment surface as a success-rate drop, because abandoned requests never log a completion. Routing across providers should be a configuration change rather than a deploy, which is the main argument for keeping routing decisions out of application code. Any serious cost reduction programme works the same way: measure per outcome, not per call.

Rollout mechanics: shadow, canary, then experiment

Shadow mode sends a copy of production traffic to the candidate and logs both outputs, serving only the incumbent. Zero user risk, and it validates schema conformance, latency, token counts and cost before anyone sees the output. It cannot tell you whether quality improved, because nobody reads the shadow output: it is a gate, not evidence.

A canary sends 1% to 5% of real traffic to the new variant with automatic rollback on a guardrail breach. It answers “is it safe to expose users to this”, which is a different question from “is it better”, and most prompt changes should stop here.

Under a flag, the variant must still come from the same hash rather than from per-request state. If a service restart reshuffles users between variants, your experiment silently becomes a per-request test with extra steps. Cache the resolved variant against the assignment key, with no TTL short enough to expire mid-session.

Roll back in under a minute, and note that this constrains architecture rather than process. Prompts must live in a versioned store a config write can point at, not inside a container image; a prompt change that needs a deploy is one you will not roll back at 02:00. Keep the previous version warm so rollback is a pointer flip, and put the prompt hash in every cache key, or a rollback will keep serving the reverted version for the whole cache TTL. See prompt management and versioning.

A concrete implementation

One event per LLM call, written where you know the outcome, with the assignment stamped on it. Not two tables joined later.

Column	Type	Why it is there
experiment_id	string	Join key, also the salt scope
variant	string	Resolved at assignment, never inferred later
unit_id	string	User or tenant id, hashed before storage if it is PII
unit_type	string	Makes the randomisation unit explicit in the data
assignment_salt	string	Lets you discard a bad randomisation without guessing
model_requested	string	The alias you asked for
model_version	string	The exact version returned, because the alias lies
prompt_hash	string	SHA-256 of the rendered prompt, not the prompt itself
params	json	temperature, top_p, max_tokens, seed
latency_ms	int	End to end, including every retry
attempts	int	Anything above 1 is a retry, and a cost event
input_tokens, output_tokens	int	Recompute cost when pricing changes
cost_usd	decimal	Provider-reported or computed, but always present
success	bool	Your task-success signal, not an HTTP 200
guardrail_breach	string array	latency, cost, refusal, safety, schema

WITH base AS (
  SELECT
    a.variant,
    a.unit_id,
    r.success,
    r.latency_ms,
    r.cost_usd
  FROM experiment_assignments AS a
  JOIN llm_requests AS r
    ON r.unit_id = a.unit_id
   AND r.experiment_id = a.experiment_id
   AND r.ts >= a.assigned_at
  WHERE a.experiment_id = 'checkout-summary-v3'
    AND a.unit_type = 'user'
    AND r.ts >= TIMESTAMP '2026-09-01'
    AND r.ts <  TIMESTAMP '2026-09-15'
),

-- One row per USER first. The unit of analysis must match the unit of
-- randomisation, otherwise correlated requests manufacture significance.
per_user AS (
  SELECT
    variant,
    unit_id,
    COUNT(*)                                      AS requests,
    AVG(CASE WHEN success THEN 1.0 ELSE 0.0 END)  AS success_rate,
    SUM(CASE WHEN success THEN 0 ELSE 1 END)      AS failures,
    SUM(cost_usd)                                 AS cost_usd,
    APPROX_QUANTILES(latency_ms, 100)[OFFSET(95)] AS user_p95_ms
  FROM base
  GROUP BY variant, unit_id
)

SELECT
  variant,
  COUNT(*)                      AS users,
  SUM(requests)                 AS requests,
  AVG(success_rate)             AS success_rate,
  SUM(cost_usd) / NULLIF(SUM(requests) - SUM(failures), 0) AS cost_per_success_usd,
  APPROX_QUANTILES(user_p95_ms, 100)[OFFSET(50)]           AS median_user_p95_ms
FROM per_user
GROUP BY variant
ORDER BY variant;

The query aggregates to one row per user and only then averages, so the unit of analysis matches the unit of randomisation. Averaging raw requests treats 20 correlated requests from one user as 20 independent observations, which understates variance and manufactures significance; at request level, use cluster-robust standard errors keyed on unit_id. Cost per success is total cost divided by successes, so failed and retried attempts are charged to the successes they eventually produced.

When an A/B test is worth it, and when it is not

Run a real A/B test only when three things hold at once. The effect is user-visible. You have enough traffic to detect the effect size you care about within two weeks. A wrong decision is expensive or hard to reverse. If any fails, ship behind a flag with a canary and a guardrail-driven automatic rollback.

• Ship behind a flag, no A/B test: latency and cost optimisations that provably do not change outputs, provider failover, caching layers, prompt compression verified by output equality on a replay set.
• Run the A/B test: a change to the system prompt of a user-facing assistant, a model swap on a revenue path, a change to how retrieved context is formatted, a change to refusal or escalation policy. These alter what users experience, and you cannot reason your way to the answer.
• Decide without an experiment, and say so: cosmetic changes with no plausible mechanism, and changes where the minimum detectable effect you can reach exceeds the effect you care about.

The failure mode to avoid is the middle: an underpowered test that runs four days, returns p = 0.04 on a secondary metric you never pre-registered, and ships because it has a p-value. That is worse than shipping behind a canary, because it launders a guess into a decision. Put the leverage in the plumbing instead: one gateway key that stamps cost, model version and prompt hash on every event, one assignment function, one event stream. The Qora API gateway is shaped around exactly that: one OpenAI-compatible endpoint with unified billing, failover and cost controls, so a new variant is a routing change rather than an integration project.

Frequently asked questions

Can I A/B test at temperature 0?

Yes, but temperature 0 does not mean deterministic in production. Greedy decoding removes sampling randomness, not batch-size-dependent kernel behaviour, mixture-of-experts routing or provider-side version changes. Identical requests can still return different tokens across calls, and will across model versions. Treat temperature 0 as low variance rather than zero variance, and keep the unit of randomisation at the user level: the variance you are controlling for is mostly traffic, not sampling.

Can I use an LLM as the judge for my primary metric?

Use it as a dense secondary signal after validating it against human labels on a held-out sample; below roughly 0.6 kappa it is too noisy to steer on. Blind the judge to variant identity and randomise candidate order to control position bias, or you are partly measuring self-preference.

What if the primary metric improves but a guardrail breaks?

The variant loses. Guardrails are non-inferiority constraints, not tie-breakers. Pre-register the bounds and treat a breach as a failure to ship, regardless of the p-value on the primary metric.

Conclusion

The decisions that matter are few and all happen before you launch: randomise by user with a hashed, salted, deterministic assignment; pick one primary metric and pre-register the cost and latency guardrails; compute the minimum detectable effect from your real traffic and refuse to run the test if the arithmetic says eleven weeks; fix the horizon; aggregate at the unit of randomisation.

Most teams get the order backwards. They launch an experiment, watch a dashboard, and reach for statistics only when a number looks interesting. The fix is not a better dashboard. It is a deterministic assignment function, an event schema carrying variant, model version and prompt hash on every call, and a rollback path that takes seconds. Build those three things and the experiment becomes cheap enough to run properly.

Related reading

• Prompt Management and Versioning in Production
• Building an LLM Eval Harness: Regression Testing for Prompts and Models
• How to Choose the Right AI Model: A Practical Model-Routing Guide
• Evaluating and Benchmarking AI Models Before You Ship