A retry is a write, not a read. Resending after a timeout asserts that the first attempt either never executed or produced no effect you care about, and against LLM APIs that is usually false. Correct retries need two mechanisms: a classifier that separates provably-unsent failures from unknown-outcome ones, and an idempotency key that collapses duplicate attempts into one execution. Backoff, jitter and budgets only stop you making an outage worse while you wait.
Why a client-side timeout tells you nothing about the server
Your HTTP client gives up at 30 seconds. The provider does not know that. For a non-streaming completion the request is already sent, the model is mid-generation, and nothing propagates your disconnect into the inference worker. The generation finishes, the output tokens are counted, and you are billed for all of them. You received nothing. In your own data this shows up as billed completion tokens with no delivered response, and it means your retry policy is paying twice.
Worked example. A model at $3.00 per million output tokens, completions averaging 1,800 tokens, a client timeout at 30 s against a real server-side latency of 34 s. One affected request costs 1,800 / 1,000,000 x $3.00 = $0.0054 and returns zero bytes. Two retries cost 2 x $0.0054 = $0.0108. At 20,000 affected requests a day that is $216 daily, roughly $6,480 a month, for nothing.
A failure taxonomy you can actually code against
Retry decisions turn on four axes: transport outcome, HTTP status, provider error type, and semantics. Broken retry code inspects one, usually the status code.
Transport failures
DNS failure, connection refused and TLS handshake failure are the only provably pre-send failures: no HTTP request was written, so nothing can have executed. Everything else is unknown-outcome. A reset before response headers is unknown: an edge proxy can accept and forward before resetting. The one question that matters: did any response byte arrive?
HTTP status classes
408, 429, 500, 502, 503 and 504 are retryable. 400, 401, 403, 404, 413 and 422 are not, because identical bytes produce an identical rejection. 401 gets one exception: a single retry after a credential refresh that changed the credential. Never treat any 5xx as retryable without reading the body.
Provider-specific overload signals
This is where generic libraries fail. OpenAI returns 429 for both rate limiting and quota exhaustion, and the two need opposite responses: rate_limit_exceeded is retryable, insufficient_quota is a billing wall that never clears. Same status, opposite decision, distinguishable only from the body’s type field. Anthropic uses 529 overloaded_error, outside the standard set, so a status in {500, 502, 503, 504} check misses it. Vertex and Gemini return RESOURCE_EXHAUSTED and UNAVAILABLE in a gRPC envelope, and some streaming paths deliver the error in-band on HTTP 200. Bedrock raises ThrottlingException.
That compounds badly: the OpenAI Python SDK retries twice by default, Anthropic’s twice, boto3’s adaptive mode more. Add an application layer with three attempts and one logical operation produces 3 x 4 = 12 upstream calls. Two retry layers multiply; they do not add.
Semantic failures
Content-filter blocks, context-length overflows, refusals and schema violations are deterministic in the input, so retrying is pure cost. At temperature above zero a filter false-positive is stochastic, and re-issuing at temperature zero can pass. That is a new request with different parameters, not a retry.
| Condition | Retryable? | Action |
|---|---|---|
| DNS failure, connection refused, TLS handshake failure | Yes, safely | Jittered retry; provably pre-send, no duplicate risk |
| Connection reset before response headers | Yes, unknown outcome | Retry only with an idempotency key |
| Read timeout after headers received | Unknown | Treat as executed; require key, log orphan tokens |
| 408 Request Timeout | Unknown | Key required; honour any Retry-After |
429 rate_limit_exceeded | Yes | Honour Retry-After, jitter, reduce concurrency |
429 insufficient_quota | No | Fail fast, page the billing owner |
| 500 Internal Server Error | Usually | Jittered retry; open breaker if sustained |
| 502 Bad Gateway | Yes | One immediate retry, then backoff |
| 503 / 504 | Yes, unknown outcome | Backoff plus key; shed load if persistent |
529 overloaded_error (Anthropic) | Yes | Long backoff, expect minutes not seconds |
ThrottlingException (Bedrock) | Yes | Disable SDK retries first, then own it |
| 400 / 422 malformed request | No | Fix the caller; retrying wastes budget |
| 401 / 403 | No, unless refreshed | Refresh credential once, then fail |
| 404 model not found | No | Fix model identifier or routing table |
| 413 payload too large | No | Truncate, chunk, or switch model |
| Content filter / refusal | No | Different prompt or parameter set, new request |
context_length_exceeded | No | Truncate or route to a longer-context model |
Exponential backoff with full jitter
Ship full jitter: delay = uniform(0, min(cap, base * 2 ** attempt)), attempt zero-indexed. Worked example, base 0.5 s, cap 20 s. Attempts 0 to 5 cap the delay at 0.5, 1, 2, 4, 8 and 16 s, with means of 0.25, 0.5, 1, 2, 4 and 8 s. Attempt 6 onward caps at min(20, 64) = 20 s, mean 10 s. A three-attempt policy adds an expected 0.25 + 0.5 = 0.75 s of sleep.
Jitter exists because failures are correlated. Without it, every client that hit the same blip retries at the same instant and the retry wave recreates the overload. Deterministic backoff also synchronises with recovery: if a provider sheds load for two seconds, every client that picked a two-second delay returns exactly as the queue drains. AWS’s published simulation of these variants found full jitter won on both request count and completion time.
Set the cap against your request budget: a 20-second sleep inside a 30-second client timeout means the retry never executes. Cap interactive calls at 20 s and batch at 60 s, and enforce a wall-clock budget per logical operation including every sleep. Honour Retry-After when present, clamped to the cap. Retries also need a budget: a token bucket of 100 tokens refilling 0.1 per successful request bounds sustained retry traffic at roughly 10% of your success rate.
import random
RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}
# Overload signals that are retryable despite non-standard codes or envelopes.
RETRYABLE_TYPES = {
"rate_limit_exceeded", "overloaded_error", "server_error",
"ThrottlingException", "RESOURCE_EXHAUSTED", "UNAVAILABLE",
}
# Signals that share a status code with a retryable error but never clear.
FATAL_TYPES = {
"insufficient_quota", "billing_hard_limit_reached", "invalid_api_key",
"invalid_request_error", "content_policy_violation",
"context_length_exceeded",
}
class RetryBudget:
"""Token bucket that caps retries at a fraction of successful traffic."""
def __init__(self, capacity: int = 100, refill_per_success: float = 0.1):
self.capacity = capacity
self.tokens = float(capacity)
self.refill = refill_per_success
def grant(self) -> bool:
if self.tokens < 1.0:
return False
self.tokens -= 1.0
return True
def on_success(self) -> None:
self.tokens = min(self.capacity, self.tokens + self.refill)
def classify(status=None, provider_type=None, exc=None, headers_received=False):
"""Return (retryable, provably_pre_send, reason)."""
if exc is not None:
if isinstance(exc, TimeoutError):
# No headers means the request may or may not have executed.
return (not headers_received), False, "timeout"
if isinstance(exc, ConnectionRefusedError) and not headers_received:
return True, True, "connection_refused"
if isinstance(exc, OSError) and not headers_received:
return True, False, "transport_error"
return False, False, "unexpected_exception"
if provider_type in FATAL_TYPES:
return False, False, "provider:" + str(provider_type)
if status in (401, 403):
return False, False, "auth"
if status is not None and 400 <= status < 500 and status not in RETRYABLE_STATUS:
return False, False, "client_error_" + str(status)
if status in RETRYABLE_STATUS or provider_type in RETRYABLE_TYPES:
return True, False, "status_" + str(status)
return False, False, "unclassified_" + str(status)
def full_jitter(attempt: int, base: float = 0.5, cap: float = 20.0) -> float:
"""attempt is zero-indexed: uniform(0, min(cap, base * 2**attempt))."""
return random.uniform(0.0, min(cap, base * (2 ** attempt)))
Idempotency keys: one key per logical operation
The key identifies intent, not an attempt. Generate it once before the first send, persist it, and reuse it byte-for-byte on every retry. Where the operation has a business identity, derive it deterministically: sha256(tenant_id + "generate_summary" + document_id + revision_id) survives restarts, deployments and queue redelivery. Where no identity exists, generate a UUIDv7 or ULID once and store it in the row that represents the operation.
Do not hash the request body: two identical prompts from different tenants are different operations, and a retry that re-serialises the message list in a different order would compute a different key. Store a body fingerprint beside the key so reuse with a changed payload is a hard error. Keep state in Redis with SET key state NX EX 86400: in_progress with a short lease so a crashed worker cannot wedge the key, completed with the response body for transparent replays, failed with the terminal error class. The TTL must exceed the worst-case client retry window.
Replay semantics must be explicit. Completed: return the stored response with its original status plus a marker header such as Idempotency-Replayed: true. In flight: 409 Conflict or 202 with a polling location, never blocking the second request on the first. Same key, different fingerprint: 422, hard fail. Terminally failed: store the error class but not the response, and let the caller re-key after fixing the input.
Provider support is uneven: some providers document an idempotency header on some endpoints, many document nothing, and support differs between streaming and non-streaming paths. Terminate idempotency in your own layer, so one component owns the dedupe table and forwards at most one upstream dispatch per key. That is the strongest argument for putting a gateway in front of providers, whether qoraapi.com or a proxy you run yourself: it becomes the single place the key is enforced. A gateway guarantees at-most-once dispatch, not at-most-once execution; if it times out upstream, the outcome is still unknown.
import { createHash, randomUUID } from "node:crypto";
type Stored = {
state: "in_progress" | "completed" | "failed";
fingerprint: string;
status?: number;
body?: unknown;
errorClass?: string;
};
const TTL_SECONDS = 86_400;
export function deriveKey(tenantId: string, operation: string, eventId: string): string {
// Stable across restarts, deployments and queue redelivery.
return createHash("sha256")
.update(`${tenantId}:${operation}:${eventId}`)
.digest("hex");
}
export function newKey(): string {
// No natural business identity: generate once, persist with the operation row.
return randomUUID();
}
function fingerprint(payload: unknown): string {
return createHash("sha256").update(JSON.stringify(payload)).digest("hex");
}
export async function withIdempotency(
redis: any,
key: string,
payload: unknown,
exec: () => Promise<{ status: number; body: unknown }>,
) {
const fp = fingerprint(payload);
const claim = await redis.set(
`idem:${key}`,
JSON.stringify({ state: "in_progress", fingerprint: fp } satisfies Stored),
"NX",
"EX",
TTL_SECONDS,
);
if (claim === null) {
const prev: Stored = JSON.parse(await redis.get(`idem:${key}`));
if (prev.fingerprint !== fp) {
return { status: 422, body: { error: "idempotency_key_reused_with_different_payload" } };
}
if (prev.state === "in_progress") {
return { status: 409, body: { error: "operation_in_progress", key } };
}
if (prev.state === "completed") {
return { status: prev.status ?? 200, body: prev.body, replayed: true };
}
return { status: 409, body: { error: "previous_attempt_failed", errorClass: prev.errorClass } };
}
try {
const result = await exec();
await redis.set(
`idem:${key}`,
JSON.stringify({ state: "completed", fingerprint: fp, ...result } satisfies Stored),
"EX",
TTL_SECONDS,
);
return result;
} catch (err: any) {
await redis.set(
`idem:${key}`,
JSON.stringify({
state: "failed",
fingerprint: fp,
errorClass: err?.code ?? "unknown",
} satisfies Stored),
"EX",
TTL_SECONDS,
);
throw err;
}
}
Idempotency for streaming responses
A partially consumed stream cannot be retried transparently, for three independent reasons. Sampling is stochastic, so a retry produces a different completion and you cannot dedupe by comparing text. Token accounting arrives in the terminal chunk, so an aborted stream leaves you with no authoritative billing number. And whether the provider keeps generating after your disconnect varies by provider.
Resume works only when the provider exposes a stable response id plus a continuation endpoint, which most do not. Last-Event-ID resumption is a property of your own SSE stream, not the provider’s token stream: you can resume delivery from a server-side buffer, but only if you kept the buffer. Restart with client-side dedupe suits machine consumers that can discard a partial result, and is wrong for chat UIs where the user already watched tokens appear. Accepting the loss means marking the message failed, regenerating, and recording the duplicate spend. What I would ship: buffer server-side, do not emit until the first chunk is committed, persist accumulated text against the key, and on retry offer an explicit regenerate action rather than a silent retry. Delivery mechanics are in AI API streaming and SSE.
Duplicate side effects are worse than duplicate spend
Money is recoverable. A sent email, a created ticket or a shipped order is not. Every tool that writes, sends or charges is non-idempotent by default, and a model that sees a tool error will call it again.
Use natural keys where a business identity exists: sha256(tenant_id + "send_invoice_email" + invoice_id) is stable across restarts, deployments and replays. Otherwise the dedupe table is the primitive that works. A unique constraint plus INSERT ... ON CONFLICT DO NOTHING with a rowcount check gives exactly one winner under concurrency; an in-process lock or a Redis GET-then-SET has a race window and fails precisely when two workers retry simultaneously.
For effects that must be atomic with a database state change, use the outbox pattern: write the intent into the outbox in the same transaction as the state change, drain it with at-least-once delivery, and make the downstream effect idempotent by passing your key onward. For irreversible external effects, use two-phase confirmation: insert a pending row carrying your key, call the downstream API with that key, then mark committed with the downstream identifier. A timeout leaves a pending row that tells you what to reconcile. Two rules for review: the key never comes from model output, and retry permission is declared by the tool author rather than inferred. Related patterns are in AI agents and tool use.
Circuit breakers, bulkheads and hedging
If every request is failing and each retries three times, you have tripled load on a system already shedding. A provider’s 429 is an instruction to reduce concurrency, not increase attempts. Aggregate retries without concurrency control are a self-inflicted denial of service.
Open the breaker on either N consecutive failures or a failure ratio above a threshold over a rolling window, with a minimum sample of 20 requests. Stay open 30 seconds initially, longer for provider-wide outages. Half-open admits exactly one probe, and the probe should be cheap, a models listing or a one-token completion rather than a real user request. If it fails, multiply the cooldown (30 s, 60 s, 120 s, capped at 5 minutes) instead of re-probing every 30 seconds.
Bulkheads are the other half: a semaphore per provider limiting in-flight requests, which retries must acquire through. If the retry path bypasses the cap, retries become the load. On 429s, halve the limit and recover at roughly 10% per minute. Use separate connection pools per provider so one provider’s slow responses cannot starve another’s sockets. The multi-provider version is in AI API failover across multiple providers.
Hedging sends a duplicate to a second provider after a latency threshold, takes the first response, and cancels the loser. It works when the tail is dominated by queueing rather than work. Set the threshold near your p95: if p50 is 1.2 s and p95 is 3.5 s, hedge at 3.5 s, so only a few percent of requests spawn a second call. It is wrong for anything with side effects, because hedging is a duplicate by design, and wrong when the slow request is slow because the model is reasoning: cancelling at 3.5 s discards a 40-second generation and you pay for the tokens already produced. Cancellation is not a refund.
Cost arithmetic. 1,000,000 daily requests, a 5% hedge rate, so 50,000 hedges. If 60% return first, you still pay for the original’s partial generation. Worst case you pay for 1,050,000 generations instead of 1,000,000: a 5% spend increase for a tail-latency win. Good trade for a user-facing chat; not for a batch job where nobody is waiting.
A retry helper that classifies, budgets, and refuses unkeyed retries
import asyncio
import time
from dataclasses import dataclass
class RetryBudgetExhausted(RuntimeError):
pass
@dataclass
class AttemptLog:
logical_id: str
attempt: int
reason: str
status: int | None
provider_type: str | None
retry_after: float | None
delay_ms: int
outcome: str
def extract_error_fields(exc):
"""Pull status, provider error type and Retry-After out of an SDK exception."""
status = getattr(exc, "status_code", None) or getattr(exc, "http_status", None)
ptype = None
body = getattr(exc, "body", None)
if isinstance(body, dict):
err = body.get("error", body)
if isinstance(err, dict):
ptype = err.get("type") or err.get("code")
if ptype is None:
ptype = getattr(exc, "code", None)
retry_after = None
headers = getattr(exc, "headers", None) or getattr(exc, "response", None)
if hasattr(headers, "get"):
raw = headers.get("retry-after")
if raw is not None:
try:
retry_after = float(raw)
except (TypeError, ValueError):
retry_after = None
return status, ptype, retry_after
async def call_with_retries(
fn,
*,
logical_id: str,
idempotency_key: str | None,
budget: RetryBudget,
max_attempts: int = 3,
base: float = 0.5,
cap: float = 20.0,
wall_clock_budget: float = 25.0,
allow_unkeyed_pre_send_retry: bool = True,
log=None,
):
"""fn is called as fn(idempotency_key). An unkeyed write is never retried
unless the failure is provably pre-send."""
if idempotency_key is None and not allow_unkeyed_pre_send_retry:
max_attempts = 1
log = log or (lambda a: None)
deadline = time.monotonic() + wall_clock_budget
last_exc = None
for attempt in range(max_attempts):
try:
result = await fn(idempotency_key)
budget.on_success()
return result
except Exception as exc: # noqa: BLE001 - classification is explicit
last_exc = exc
status, ptype, retry_after = extract_error_fields(exc)
headers_received = status is not None
retryable, pre_send, reason = classify(status, ptype, exc, headers_received)
if not retryable:
raise
if attempt == max_attempts - 1:
raise
if idempotency_key is None and not pre_send:
# Unknown outcome with no key: retrying risks a duplicate effect.
raise
if not budget.grant():
raise RetryBudgetExhausted(reason) from exc
delay = retry_after if retry_after is not None else full_jitter(attempt, base, cap)
delay = min(delay, cap)
if time.monotonic() + delay > deadline:
raise
log(
AttemptLog(
logical_id=logical_id,
attempt=attempt,
reason=reason,
status=status,
provider_type=ptype,
retry_after=retry_after,
delay_ms=int(delay * 1000),
outcome="retry",
)
)
await asyncio.sleep(delay)
raise last_exc
Three properties make this correct. Classification happens before any retry decision, so a 429 carrying insufficient_quota never consumes budget and never adds load. The budget is checked before sleeping, so amplification is bounded even when everything is failing. And the unkeyed case is gated on pre_send, so a non-idempotent call retries only when the classifier can prove no bytes reached the provider. Pair it with max_retries=0 on the SDK so exactly one retry layer exists.
What to log per attempt
Log one record per attempt, not per logical request. The fields that earn their keep: logical request id, attempt number, idempotency key, provider, model, endpoint, error class, HTTP status, provider error type, the provider’s own request id from the x-request-id or request-id header, any Retry-After, computed delay, prompt and completion tokens, cost, duration, and outcome. That provider request id is your only handle in a billing dispute.
From those records derive five metrics: retry ratio (attempts divided by logical operations), amplification (upstream calls divided by logical operations), retry success ratio, duplicate effects served from the dedupe table, and orphaned tokens, meaning billed tokens with no delivered response. These separate two situations that look identical in a success-rate dashboard. One flaky request: low retry ratio, amplification near 1.0x, high retry success ratio, normal latency. A masked outage: climbing retry ratio and amplification, falling retry success ratio, and rising p50 and p99 on requests that ultimately succeed, because backoff sleeps sit inside the request. Your error rate can look flat while the service is three times slower: that is the state worth alerting on. Metric design for this layer is covered in LLM observability.
The default retry policy I would ship
- Exactly one retry layer. Set SDK retries to zero and own the policy centrally, so amplification is your policy rather than the product of two.
- Retry on connection refused, DNS failure, TLS failure, 408, 429 rate-limit, 500, 502, 503, 504, and provider overload types (529
overloaded_error,ThrottlingException,RESOURCE_EXHAUSTED,UNAVAILABLE). - Never retry 400, 403, 404, 413, 422, content-filter blocks, context-length overflows, or quota and billing errors. Allow one retry on 401, only after a credential refresh that changed the credential.
- Three attempts for interactive calls, five for batch. Full jitter, base 500 ms, cap 20 s interactive and 60 s batch, with a 25 s wall-clock budget per logical operation including every sleep.
- Honour
Retry-Afterwhenever present, clamped to the cap. - An idempotency key is mandatory for every request that can bill or cause a side effect. Without a key, at most one retry, and only for provably pre-send failures.
- A retry budget token bucket of 100 tokens refilling at 0.1 per successful logical request, checked before every sleep.
- One breaker per provider: 20-request minimum sample, open at 50% failures, 30 s cooldown with exponential backoff capped at 5 minutes, one cheap half-open probe.
- One in-flight semaphore per provider, which retries must acquire through. Halve the limit on 429s, recover at roughly 10% per minute.
- Never retry a partially consumed stream. Never hedge a non-idempotent call. Never let a model generate an idempotency key.
That policy is deliberately conservative about retrying and aggressive about classifying. Most teams I review have the ratio inverted: a generic retry decorator everywhere, no classification, no key. The result looks resilient in staging and doubles the inference bill during the first real incident.
Frequently asked questions
Is a 500 always safe to retry?
No. A 500 can arrive after the request was fully processed and the response failed on the way out, which makes it unknown-outcome rather than pre-send. Some gateways also return 500 wrapping an upstream 400 or a content-filter rejection. Treat 500 as retryable only when the body’s error type indicates a server-side fault, and require a key.
Should the idempotency key be a UUID?
A UUID is fine for uniqueness but not sufficient. The key must be generated once per logical operation and persisted before the first send. A UUIDv4 generated inside the function that makes the HTTP call is regenerated on every retry, which makes it useless. Prefer UUIDv7 or ULID for sortability, and prefer derivation from a business event id.
Can I just retry the whole agent run instead of individual steps?
No. An agent run is a sequence of steps, some with side effects, so retrying the run re-executes every completed step and duplicates each effect. Retry at the step level with a key derived from the run id and step index, and treat the run as a state machine that resumes from the last committed step. A step that cannot be made idempotent needs a compensating action, not a retry.
Does my provider deduplicate retries automatically?
Assume not. Idempotency header support varies by provider, by endpoint, and often between streaming and non-streaming paths. Even where a header exists, verify it experimentally, because a silently ignored header looks exactly like a working one until you inspect the bill.
How long should the idempotency record live?
Longer than the worst-case window in which a client could replay: your maximum client timeout multiplied by your maximum attempts, plus queue delay, plus clock skew. A 30 s timeout with three attempts and 60 s of queueing gives roughly 150 s. Twenty-four hours of headroom is conventional because it also covers a restarted worker or a stuck batch job.
Conclusion
Retry correctness is a classification problem first and a timing problem second. Decide whether an attempt could have executed before deciding how long to wait, persist one idempotency key per logical operation before the first byte leaves, and cap the blast radius with a budget, a breaker and a per-provider semaphore. The backoff formula is the easy part; the classifier and the key are where duplicate generations, duplicate tool calls and double charges are actually prevented. If you would rather not maintain the dedupe table, the classification rules and the breaker state yourself, that is precisely the layer a gateway such as qoraapi.com can own, provided you understand that it bounds duplicate dispatch and not duplicate execution. Start with three attempts, full jitter, a mandatory key on every write, and an alert on amplification above 1.2.
Leave a Reply