{"id":298,"date":"2026-09-22T17:47:54","date_gmt":"2026-09-22T09:47:54","guid":{"rendered":"https:\/\/wp.qoraapi.com\/idempotency-safe-retries-ai-api\/"},"modified":"2026-09-22T17:49:29","modified_gmt":"2026-09-22T09:49:29","slug":"idempotency-safe-retries-ai-api","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/idempotency-safe-retries-ai-api\/","title":{"rendered":"Idempotency and Safe Retries for AI APIs"},"content":{"rendered":"\n

A retry is a write, not a read. Resending after a timeout asserts that the first attempt either never executed or produced no effect you care about, and against LLM APIs that is usually false. Correct retries need two mechanisms: a classifier that separates provably-unsent failures from unknown-outcome ones, and an idempotency key that collapses duplicate attempts into one execution. Backoff, jitter and budgets only stop you making an outage worse while you wait.<\/p>\n\n\n\n

Why a client-side timeout tells you nothing about the server<\/h2>\n\n\n\n

Your HTTP client gives up at 30 seconds. The provider does not know that. For a non-streaming completion the request is already sent, the model is mid-generation, and nothing propagates your disconnect into the inference worker. The generation finishes, the output tokens are counted, and you are billed for all of them. You received nothing. In your own data this shows up as billed completion tokens with no delivered response, and it means your retry policy is paying twice.<\/p>\n\n\n\n

Worked example. A model at $3.00 per million output tokens, completions averaging 1,800 tokens, a client timeout at 30 s against a real server-side latency of 34 s. One affected request costs 1,800 \/ 1,000,000 x $3.00 = $0.0054 and returns zero bytes. Two retries cost 2 x $0.0054 = $0.0108. At 20,000 affected requests a day that is $216 daily, roughly $6,480 a month, for nothing.<\/p>\n\n\n\n

A failure taxonomy you can actually code against<\/h2>\n\n\n\n

Retry decisions turn on four axes: transport outcome, HTTP status, provider error type, and semantics. Broken retry code inspects one, usually the status code.<\/p>\n\n\n\n

Transport failures<\/h3>\n\n\n\n

DNS failure, connection refused and TLS handshake failure are the only provably pre-send failures: no HTTP request was written, so nothing can have executed. Everything else is unknown-outcome. A reset before response headers is unknown: an edge proxy can accept and forward before resetting. The one question that matters: did any response byte arrive?<\/p>\n\n\n\n

HTTP status classes<\/h3>\n\n\n\n

408, 429, 500, 502, 503 and 504 are retryable. 400, 401, 403, 404, 413 and 422 are not, because identical bytes produce an identical rejection. 401 gets one exception: a single retry after a credential refresh that changed the credential. Never treat any 5xx as retryable without reading the body.<\/p>\n\n\n\n

Provider-specific overload signals<\/h3>\n\n\n\n

This is where generic libraries fail. OpenAI returns 429 for both rate limiting and quota exhaustion, and the two need opposite responses: rate_limit_exceeded<\/code> is retryable, insufficient_quota<\/code> is a billing wall that never clears. Same status, opposite decision, distinguishable only from the body’s type<\/code> field. Anthropic uses 529 overloaded_error<\/code>, outside the standard set, so a status in {500, 502, 503, 504}<\/code> check misses it. Vertex and Gemini return RESOURCE_EXHAUSTED<\/code> and UNAVAILABLE<\/code> in a gRPC envelope, and some streaming paths deliver the error in-band on HTTP 200. Bedrock raises ThrottlingException<\/code>.<\/p>\n\n\n\n

That compounds badly: the OpenAI Python SDK retries twice by default, Anthropic’s twice, boto3’s adaptive mode more. Add an application layer with three attempts and one logical operation produces 3 x 4 = 12 upstream calls. Two retry layers multiply; they do not add.<\/p>\n\n\n\n

Semantic failures<\/h3>\n\n\n\n

Content-filter blocks, context-length overflows, refusals and schema violations are deterministic in the input, so retrying is pure cost. At temperature above zero a filter false-positive is stochastic, and re-issuing at temperature zero can pass. That is a new request with different parameters, not a retry.<\/p>\n\n\n\n

Condition<\/th>Retryable?<\/th>Action<\/th><\/tr><\/thead>
DNS failure, connection refused, TLS handshake failure<\/td>Yes, safely<\/td>Jittered retry; provably pre-send, no duplicate risk<\/td><\/tr>
Connection reset before response headers<\/td>Yes, unknown outcome<\/td>Retry only with an idempotency key<\/td><\/tr>
Read timeout after headers received<\/td>Unknown<\/td>Treat as executed; require key, log orphan tokens<\/td><\/tr>
408 Request Timeout<\/td>Unknown<\/td>Key required; honour any Retry-After<\/td><\/tr>
429 rate_limit_exceeded<\/code><\/td>Yes<\/td>Honour Retry-After, jitter, reduce concurrency<\/td><\/tr>
429 insufficient_quota<\/code><\/td>No<\/td>Fail fast, page the billing owner<\/td><\/tr>
500 Internal Server Error<\/td>Usually<\/td>Jittered retry; open breaker if sustained<\/td><\/tr>
502 Bad Gateway<\/td>Yes<\/td>One immediate retry, then backoff<\/td><\/tr>
503 \/ 504<\/td>Yes, unknown outcome<\/td>Backoff plus key; shed load if persistent<\/td><\/tr>
529 overloaded_error<\/code> (Anthropic)<\/td>Yes<\/td>Long backoff, expect minutes not seconds<\/td><\/tr>
ThrottlingException<\/code> (Bedrock)<\/td>Yes<\/td>Disable SDK retries first, then own it<\/td><\/tr>
400 \/ 422 malformed request<\/td>No<\/td>Fix the caller; retrying wastes budget<\/td><\/tr>
401 \/ 403<\/td>No, unless refreshed<\/td>Refresh credential once, then fail<\/td><\/tr>
404 model not found<\/td>No<\/td>Fix model identifier or routing table<\/td><\/tr>
413 payload too large<\/td>No<\/td>Truncate, chunk, or switch model<\/td><\/tr>
Content filter \/ refusal<\/td>No<\/td>Different prompt or parameter set, new request<\/td><\/tr>
context_length_exceeded<\/code><\/td>No<\/td>Truncate or route to a longer-context model<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Exponential backoff with full jitter<\/h2>\n\n\n\n

Ship full jitter: delay = uniform(0, min(cap, base * 2 ** attempt))<\/code>, attempt zero-indexed. Worked example, base 0.5 s, cap 20 s. Attempts 0 to 5 cap the delay at 0.5, 1, 2, 4, 8 and 16 s, with means of 0.25, 0.5, 1, 2, 4 and 8 s. Attempt 6 onward caps at min(20, 64) = 20 s, mean 10 s. A three-attempt policy adds an expected 0.25 + 0.5 = 0.75 s of sleep.<\/p>\n\n\n\n

Jitter exists because failures are correlated. Without it, every client that hit the same blip retries at the same instant and the retry wave recreates the overload. Deterministic backoff also synchronises with recovery: if a provider sheds load for two seconds, every client that picked a two-second delay returns exactly as the queue drains. AWS’s published simulation of these variants found full jitter won on both request count and completion time.<\/p>\n\n\n\n

Set the cap against your request budget: a 20-second sleep inside a 30-second client timeout means the retry never executes. Cap interactive calls at 20 s and batch at 60 s, and enforce a wall-clock budget per logical operation including every sleep. Honour Retry-After<\/code> when present, clamped to the cap. Retries also need a budget: a token bucket of 100 tokens refilling 0.1 per successful request bounds sustained retry traffic at roughly 10% of your success rate.<\/p>\n\n\n\n

import random\n\nRETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}\n\n# Overload signals that are retryable despite non-standard codes or envelopes.\nRETRYABLE_TYPES = {\n    \"rate_limit_exceeded\", \"overloaded_error\", \"server_error\",\n    \"ThrottlingException\", \"RESOURCE_EXHAUSTED\", \"UNAVAILABLE\",\n}\n\n# Signals that share a status code with a retryable error but never clear.\nFATAL_TYPES = {\n    \"insufficient_quota\", \"billing_hard_limit_reached\", \"invalid_api_key\",\n    \"invalid_request_error\", \"content_policy_violation\",\n    \"context_length_exceeded\",\n}\n\nclass RetryBudget:\n    \"\"\"Token bucket that caps retries at a fraction of successful traffic.\"\"\"\n\n    def __init__(self, capacity: int = 100, refill_per_success: float = 0.1):\n        self.capacity = capacity\n        self.tokens = float(capacity)\n        self.refill = refill_per_success\n\n    def grant(self) -> bool:\n        if self.tokens < 1.0:\n            return False\n        self.tokens -= 1.0\n        return True\n\n    def on_success(self) -> None:\n        self.tokens = min(self.capacity, self.tokens + self.refill)\n\ndef classify(status=None, provider_type=None, exc=None, headers_received=False):\n    \"\"\"Return (retryable, provably_pre_send, reason).\"\"\"\n    if exc is not None:\n        if isinstance(exc, TimeoutError):\n            # No headers means the request may or may not have executed.\n            return (not headers_received), False, \"timeout\"\n        if isinstance(exc, ConnectionRefusedError) and not headers_received:\n            return True, True, \"connection_refused\"\n        if isinstance(exc, OSError) and not headers_received:\n            return True, False, \"transport_error\"\n        return False, False, \"unexpected_exception\"\n\n    if provider_type in FATAL_TYPES:\n        return False, False, \"provider:\" + str(provider_type)\n    if status in (401, 403):\n        return False, False, \"auth\"\n    if status is not None and 400 <= status < 500 and status not in RETRYABLE_STATUS:\n        return False, False, \"client_error_\" + str(status)\n    if status in RETRYABLE_STATUS or provider_type in RETRYABLE_TYPES:\n        return True, False, \"status_\" + str(status)\n    return False, False, \"unclassified_\" + str(status)\n\ndef full_jitter(attempt: int, base: float = 0.5, cap: float = 20.0) -> float:\n    \"\"\"attempt is zero-indexed: uniform(0, min(cap, base * 2**attempt)).\"\"\"\n    return random.uniform(0.0, min(cap, base * (2 ** attempt)))\n<\/code><\/pre>\n\n\n\n
Idempotency keys: one key per logical operation<\/h2>\n\n\n\n
The key identifies intent, not an attempt. Generate it once before the first send, persist it, and reuse it byte-for-byte on every retry. Where the operation has a business identity, derive it deterministically: sha256(tenant_id + \"generate_summary\" + document_id + revision_id)<\/code> survives restarts, deployments and queue redelivery. Where no identity exists, generate a UUIDv7 or ULID once and store it in the row that represents the operation.<\/p>\n\n\n\n
Do not hash the request body: two identical prompts from different tenants are different operations, and a retry that re-serialises the message list in a different order would compute a different key. Store a body fingerprint beside the key so reuse with a changed payload is a hard error. Keep state in Redis with SET key state NX EX 86400<\/code>: in_progress<\/code> with a short lease so a crashed worker cannot wedge the key, completed<\/code> with the response body for transparent replays, failed<\/code> with the terminal error class. The TTL must exceed the worst-case client retry window.<\/p>\n\n\n\n
Replay semantics must be explicit. Completed: return the stored response with its original status plus a marker header such as Idempotency-Replayed: true<\/code>. In flight: 409 Conflict or 202 with a polling location, never blocking the second request on the first. Same key, different fingerprint: 422, hard fail. Terminally failed: store the error class but not the response, and let the caller re-key after fixing the input.<\/p>\n\n\n\n
Provider support is uneven: some providers document an idempotency header on some endpoints, many document nothing, and support differs between streaming and non-streaming paths. Terminate idempotency in your own layer, so one component owns the dedupe table and forwards at most one upstream dispatch per key. That is the strongest argument for putting a gateway in front of providers, whether qoraapi.com<\/a> or a proxy you run yourself: it becomes the single place the key is enforced. A gateway guarantees at-most-once dispatch, not at-most-once execution; if it times out upstream, the outcome is still unknown.<\/p>\n\n\n\n
import { createHash, randomUUID } from \"node:crypto\";\n\ntype Stored = {\n  state: \"in_progress\" | \"completed\" | \"failed\";\n  fingerprint: string;\n  status?: number;\n  body?: unknown;\n  errorClass?: string;\n};\n\nconst TTL_SECONDS = 86_400;\n\nexport function deriveKey(tenantId: string, operation: string, eventId: string): string {\n  \/\/ Stable across restarts, deployments and queue redelivery.\n  return createHash(\"sha256\")\n    .update(`${tenantId}:${operation}:${eventId}`)\n    .digest(\"hex\");\n}\n\nexport function newKey(): string {\n  \/\/ No natural business identity: generate once, persist with the operation row.\n  return randomUUID();\n}\n\nfunction fingerprint(payload: unknown): string {\n  return createHash(\"sha256\").update(JSON.stringify(payload)).digest(\"hex\");\n}\n\nexport async function withIdempotency(\n  redis: any,\n  key: string,\n  payload: unknown,\n  exec: () => Promise<{ status: number; body: unknown }>,\n) {\n  const fp = fingerprint(payload);\n  const claim = await redis.set(\n    `idem:${key}`,\n    JSON.stringify({ state: \"in_progress\", fingerprint: fp } satisfies Stored),\n    \"NX\",\n    \"EX\",\n    TTL_SECONDS,\n  );\n\n  if (claim === null) {\n    const prev: Stored = JSON.parse(await redis.get(`idem:${key}`));\n\n    if (prev.fingerprint !== fp) {\n      return { status: 422, body: { error: \"idempotency_key_reused_with_different_payload\" } };\n    }\n    if (prev.state === \"in_progress\") {\n      return { status: 409, body: { error: \"operation_in_progress\", key } };\n    }\n    if (prev.state === \"completed\") {\n      return { status: prev.status ?? 200, body: prev.body, replayed: true };\n    }\n    return { status: 409, body: { error: \"previous_attempt_failed\", errorClass: prev.errorClass } };\n  }\n\n  try {\n    const result = await exec();\n    await redis.set(\n      `idem:${key}`,\n      JSON.stringify({ state: \"completed\", fingerprint: fp, ...result } satisfies Stored),\n      \"EX\",\n      TTL_SECONDS,\n    );\n    return result;\n  } catch (err: any) {\n    await redis.set(\n      `idem:${key}`,\n      JSON.stringify({\n        state: \"failed\",\n        fingerprint: fp,\n        errorClass: err?.code ?? \"unknown\",\n      } satisfies Stored),\n      \"EX\",\n      TTL_SECONDS,\n    );\n    throw err;\n  }\n}\n<\/code><\/pre>\n\n\n\n
Idempotency for streaming responses<\/h2>\n\n\n\n
A partially consumed stream cannot be retried transparently, for three independent reasons. Sampling is stochastic, so a retry produces a different completion and you cannot dedupe by comparing text. Token accounting arrives in the terminal chunk, so an aborted stream leaves you with no authoritative billing number. And whether the provider keeps generating after your disconnect varies by provider.<\/p>\n\n\n\n
Resume works only when the provider exposes a stable response id plus a continuation endpoint, which most do not. Last-Event-ID<\/code> resumption is a property of your own SSE stream, not the provider’s token stream: you can resume delivery from a server-side buffer, but only if you kept the buffer. Restart with client-side dedupe suits machine consumers that can discard a partial result, and is wrong for chat UIs where the user already watched tokens appear. Accepting the loss means marking the message failed, regenerating, and recording the duplicate spend. What I would ship: buffer server-side, do not emit until the first chunk is committed, persist accumulated text against the key, and on retry offer an explicit regenerate action rather than a silent retry. Delivery mechanics are in AI API streaming and SSE<\/a>.<\/p>\n\n\n\n
Duplicate side effects are worse than duplicate spend<\/h2>\n\n\n\n
Money is recoverable. A sent email, a created ticket or a shipped order is not. Every tool that writes, sends or charges is non-idempotent by default, and a model that sees a tool error will call it again.<\/p>\n\n\n\n
Use natural keys where a business identity exists: sha256(tenant_id + \"send_invoice_email\" + invoice_id)<\/code> is stable across restarts, deployments and replays. Otherwise the dedupe table is the primitive that works. A unique constraint plus INSERT ... ON CONFLICT DO NOTHING<\/code> with a rowcount check gives exactly one winner under concurrency; an in-process lock or a Redis GET-then-SET has a race window and fails precisely when two workers retry simultaneously.<\/p>\n\n\n\n
For effects that must be atomic with a database state change, use the outbox pattern: write the intent into the outbox in the same transaction as the state change, drain it with at-least-once delivery, and make the downstream effect idempotent by passing your key onward. For irreversible external effects, use two-phase confirmation: insert a pending<\/code> row carrying your key, call the downstream API with that key, then mark committed<\/code> with the downstream identifier. A timeout leaves a pending row that tells you what to reconcile. Two rules for review: the key never comes from model output, and retry permission is declared by the tool author rather than inferred. Related patterns are in AI agents and tool use<\/a>.<\/p>\n\n\n\n
Circuit breakers, bulkheads and hedging<\/h2>\n\n\n\n
If every request is failing and each retries three times, you have tripled load on a system already shedding. A provider’s 429 is an instruction to reduce concurrency, not increase attempts. Aggregate retries without concurrency control are a self-inflicted denial of service.<\/p>\n\n\n\n
Open the breaker on either N consecutive failures or a failure ratio above a threshold over a rolling window, with a minimum sample of 20 requests. Stay open 30 seconds initially, longer for provider-wide outages. Half-open admits exactly one probe, and the probe should be cheap, a models listing or a one-token completion rather than a real user request. If it fails, multiply the cooldown (30 s, 60 s, 120 s, capped at 5 minutes) instead of re-probing every 30 seconds.<\/p>\n\n\n\n
Bulkheads are the other half: a semaphore per provider limiting in-flight requests, which retries must acquire through. If the retry path bypasses the cap, retries become the load. On 429s, halve the limit and recover at roughly 10% per minute. Use separate connection pools per provider so one provider’s slow responses cannot starve another’s sockets. The multi-provider version is in AI API failover across multiple providers<\/a>.<\/p>\n\n\n\n
Hedging sends a duplicate to a second provider after a latency threshold, takes the first response, and cancels the loser. It works when the tail is dominated by queueing rather than work. Set the threshold near your p95: if p50 is 1.2 s and p95 is 3.5 s, hedge at 3.5 s, so only a few percent of requests spawn a second call. It is wrong for anything with side effects, because hedging is a duplicate by design, and wrong when the slow request is slow because the model is reasoning: cancelling at 3.5 s discards a 40-second generation and you pay for the tokens already produced. Cancellation is not a refund.<\/p>\n\n\n\n
Cost arithmetic. 1,000,000 daily requests, a 5% hedge rate, so 50,000 hedges. If 60% return first, you still pay for the original’s partial generation. Worst case you pay for 1,050,000 generations instead of 1,000,000: a 5% spend increase for a tail-latency win. Good trade for a user-facing chat; not for a batch job where nobody is waiting.<\/p>\n\n\n\n
A retry helper that classifies, budgets, and refuses unkeyed retries<\/h2>\n\n\n\n
import asyncio\nimport time\nfrom dataclasses import dataclass\n\nclass RetryBudgetExhausted(RuntimeError):\n    pass\n\n@dataclass\nclass AttemptLog:\n    logical_id: str\n    attempt: int\n    reason: str\n    status: int | None\n    provider_type: str | None\n    retry_after: float | None\n    delay_ms: int\n    outcome: str\n\ndef extract_error_fields(exc):\n    \"\"\"Pull status, provider error type and Retry-After out of an SDK exception.\"\"\"\n    status = getattr(exc, \"status_code\", None) or getattr(exc, \"http_status\", None)\n    ptype = None\n    body = getattr(exc, \"body\", None)\n    if isinstance(body, dict):\n        err = body.get(\"error\", body)\n        if isinstance(err, dict):\n            ptype = err.get(\"type\") or err.get(\"code\")\n    if ptype is None:\n        ptype = getattr(exc, \"code\", None)\n\n    retry_after = None\n    headers = getattr(exc, \"headers\", None) or getattr(exc, \"response\", None)\n    if hasattr(headers, \"get\"):\n        raw = headers.get(\"retry-after\")\n        if raw is not None:\n            try:\n                retry_after = float(raw)\n            except (TypeError, ValueError):\n                retry_after = None\n    return status, ptype, retry_after\n\nasync def call_with_retries(\n    fn,\n    *,\n    logical_id: str,\n    idempotency_key: str | None,\n    budget: RetryBudget,\n    max_attempts: int = 3,\n    base: float = 0.5,\n    cap: float = 20.0,\n    wall_clock_budget: float = 25.0,\n    allow_unkeyed_pre_send_retry: bool = True,\n    log=None,\n):\n    \"\"\"fn is called as fn(idempotency_key). An unkeyed write is never retried\n    unless the failure is provably pre-send.\"\"\"\n    if idempotency_key is None and not allow_unkeyed_pre_send_retry:\n        max_attempts = 1\n\n    log = log or (lambda a: None)\n    deadline = time.monotonic() + wall_clock_budget\n    last_exc = None\n\n    for attempt in range(max_attempts):\n        try:\n            result = await fn(idempotency_key)\n            budget.on_success()\n            return result\n        except Exception as exc:  # noqa: BLE001 - classification is explicit\n            last_exc = exc\n            status, ptype, retry_after = extract_error_fields(exc)\n            headers_received = status is not None\n            retryable, pre_send, reason = classify(status, ptype, exc, headers_received)\n\n            if not retryable:\n                raise\n            if attempt == max_attempts - 1:\n                raise\n            if idempotency_key is None and not pre_send:\n                # Unknown outcome with no key: retrying risks a duplicate effect.\n                raise\n            if not budget.grant():\n                raise RetryBudgetExhausted(reason) from exc\n\n            delay = retry_after if retry_after is not None else full_jitter(attempt, base, cap)\n            delay = min(delay, cap)\n            if time.monotonic() + delay > deadline:\n                raise\n\n            log(\n                AttemptLog(\n                    logical_id=logical_id,\n                    attempt=attempt,\n                    reason=reason,\n                    status=status,\n                    provider_type=ptype,\n                    retry_after=retry_after,\n                    delay_ms=int(delay * 1000),\n                    outcome=\"retry\",\n                )\n            )\n            await asyncio.sleep(delay)\n\n    raise last_exc\n<\/code><\/pre>\n\n\n\n
Three properties make this correct. Classification happens before any retry decision, so a 429 carrying insufficient_quota<\/code> never consumes budget and never adds load. The budget is checked before sleeping, so amplification is bounded even when everything is failing. And the unkeyed case is gated on pre_send<\/code>, so a non-idempotent call retries only when the classifier can prove no bytes reached the provider. Pair it with max_retries=0<\/code> on the SDK so exactly one retry layer exists.<\/p>\n\n\n\n
What to log per attempt<\/h2>\n\n\n\n
Log one record per attempt, not per logical request. The fields that earn their keep: logical request id, attempt number, idempotency key, provider, model, endpoint, error class, HTTP status, provider error type, the provider’s own request id from the x-request-id<\/code> or request-id<\/code> header, any Retry-After<\/code>, computed delay, prompt and completion tokens, cost, duration, and outcome. That provider request id is your only handle in a billing dispute.<\/p>\n\n\n\n
From those records derive five metrics: retry ratio (attempts divided by logical operations), amplification (upstream calls divided by logical operations), retry success ratio, duplicate effects served from the dedupe table, and orphaned tokens, meaning billed tokens with no delivered response. These separate two situations that look identical in a success-rate dashboard. One flaky request: low retry ratio, amplification near 1.0x, high retry success ratio, normal latency. A masked outage: climbing retry ratio and amplification, falling retry success ratio, and rising p50 and p99 on requests that ultimately succeed, because backoff sleeps sit inside the request. Your error rate can look flat while the service is three times slower: that is the state worth alerting on. Metric design for this layer is covered in LLM observability<\/a>.<\/p>\n\n\n\n
The default retry policy I would ship<\/h2>\n\n\n\n