Async AI APIs: Job Queues, Webhooks, Long Tasks

Synchronous request/response stops working the moment a model call can outlive the infrastructure in front of it. A 60-second load-balancer idle timeout, a 29-second serverless integration limit, or a mobile client that switches networks mid-request will kill a generation that is running fine on the provider side. Those limits are not yours to raise, so decouple submission from completion: the client gets a durable job id immediately, and the result arrives later by poll or webhook.

Why synchronous request/response breaks down

Every layer between your user and the model has its own clock, and the smallest one wins. nginx defaults proxy_read_timeout to 60 seconds. An AWS ALB idles connections out at 60 seconds. API Gateway enforces a hard 29-second integration timeout you cannot raise. None of them knows the generation is healthy; they terminate on wall-clock time. Serverless sharpens the mismatch: a Lambda can run for 15 minutes, but the front door in front of it caps at 29 seconds.

Mobile and unreliable clients

A phone that locks its screen, changes networks, or gets backgrounded drops the socket. The work is gone from the client’s perspective even though you already paid for it. Resuming requires a server-side identity that a synchronous endpoint cannot provide.

Generations that are legitimately slow

Summarising a 300-page contract, transcribing an hour of audio, generating video, or running an extended-reasoning model through a hard problem takes minutes. No prompt-engineering trick compresses a 90-second generation into a 30-second window. Batch pipelines hit this immediately: the larger the batch, the more certain some item blows the budget.

Three delivery patterns compared

There are exactly three shapes you can give an asynchronous-capable API, and each is correct under different constraints.

Synchronous with streaming. One connection, tokens arriving as SSE chunks. Low time-to-first-token, unbounded duration, and a dropped connection loses the remainder unless the server supports resumption.
Submit-and-poll. The POST returns 202 Accepted with a job id; the client polls GET /jobs/{id} until terminal. Trivial to implement and works from cron and a CLI.
Submit-with-webhook. The server pushes the result to a URL you registered. Lowest latency, largest operational surface.

Pattern	Client complexity	Delivery latency	Failure behaviour	Works from	Best for
Synchronous + streaming	Low	Low to first token; unbounded to completion	Connection drop loses remaining output unless resumable	Any HTTP client, including browsers	Interactive chat, short completions
Submit-and-poll	Medium: retry and backoff logic in the client	Bounded by your poll interval	Client can always resume; state lives server-side	Any HTTP client, including CLI and cron	Batch work, internal tools, low volume, no public endpoint
Submit-with-webhook	High: public endpoint, HMAC verification, deduplication	Lowest: pushed the moment the job completes	Delivery depends on your endpoint being up; needs retries and a reconciliation sweep	Servers you operate	High-volume, user-facing long jobs

These are not mutually exclusive: a chat product streams, a document pipeline submits and webhooks, an internal report polls. Standardising on one shape is the mistake. An API gateway is the natural place to normalise the job contract so clients never learn which provider ran the work.

The job lifecycle state machine

Model the job explicitly rather than inferring it from a queue message. Six states are enough.

queued — accepted and durable, not yet claimed.
running — a specific worker holds a lease and is accountable for it.
succeeded — terminal; the result is persisted and retrievable.
failed — terminal; the error is recorded, no further attempts.
cancelled — terminal; requested by the client.
expired — terminal; outlived its deadline or its retention window.

Persist every transition

Write each transition to an append-only job_events table as well as updating current state on the job row. The row answers “where is it now” with one indexed read; the log answers “what actually happened” during an incident. Same split as any observability pipeline.

Why “running” must have a lease

A running state with no owner is a latent bug. If a worker crashes mid-call, a naive implementation leaves the job running forever: no retry, no alert, nothing for a reaper to match. Define running as “worker W holds a lease expiring at T”, renewed by a heartbeat, and a reaper reclaims any row where lease_expires_at < now().

Lease length is a real trade-off. Too short and a slow-but-healthy job is reclaimed and run twice; too long and a crashed worker blocks it for the full duration. A 60-second lease with a 15-second heartbeat tolerates three missed heartbeats, absorbing a GC pause without sluggish recovery.

Queue design

Choosing a broker

You probably do not need Kafka. For tens of thousands of jobs per day, Postgres with SELECT ... FOR UPDATE SKIP LOCKED is a complete transactional queue and removes a piece of infrastructure. It also lets the job row and the queue entry commit in the same transaction, which is the property you want.

Broker	Delivery semantics	Visibility timeout	Best fit
Postgres SKIP LOCKED	At-least-once	A column you manage yourself	Under 1M jobs/day; transactional enqueue, no extra infrastructure
Redis Streams	At-least-once	Pending entries list plus XCLAIM	High throughput, low latency, consumer groups
SQS	At-least-once; FIFO adds exactly-once processing within the queue	Native, extendable up to 12 hours	Managed, AWS-native, native dead-letter redrive
RabbitMQ	At-least-once	Native, per-consumer	Routing rules, priorities, per-message TTL

At-least-once is what you get

End-to-end exactly-once delivery does not exist. What you can build is exactly-once effects: at-least-once delivery plus idempotent consumers. SQS FIFO offers exactly-once processing within a queue via a five-minute dedup window, but that guarantee stops at the queue boundary.

Visibility timeout

A visibility timeout is the window during which a claimed message is hidden from other consumers. If it expires before you finish, the message is redelivered and a second worker starts the same job. It must exceed your p99 processing time, or the worker must extend it. SQS exposes ChangeMessageVisibility; on Postgres, the lease heartbeat.

Dead-letter queues

After N failed receives, move the message to a DLQ instead of retrying forever. A poison message that fails deterministically — malformed payload, retired model, empty account — otherwise consumes worker capacity indefinitely. Alarm on DLQ depth: an empty DLQ is healthy, one growing for an hour is an incident.

Never call the provider inside the transaction

This is the most common design error in job systems. The pattern looks reasonable: begin a transaction, insert the job row, call the model, update to succeeded, commit. It is wrong for three reasons.

It holds a transaction open for the entire external call. Connection pools are small, and a handful of slow generations exhausts them, stalling unrelated parts of the application.
If the call succeeds but the transaction rolls back, you have paid for a generation and thrown the result away.
If the transaction commits but the call fails, you have a job row asserting success with no result behind it.

Separate them. Commit the job row and an outbox event in one short transaction, then have a dispatcher publish after that commit. The outbox pattern makes persist-and-enqueue atomic, at the cost of a small bounded delay.

Webhook consumer correctness

A webhook endpoint is an unauthenticated public write path into your system. Treat it with the suspicion you would apply to any other one.

Verify the signature over the raw body

Compute the HMAC over the exact bytes you received, before JSON parsing or re-serialisation. Most verification bugs come from hashing a re-serialised object whose key order or whitespace no longer matches. Read the raw buffer, verify, then parse, and compare in constant time.

Reject replays with a timestamp window

Sign the timestamp with the payload and reject anything outside a tolerance window — five minutes is practical. Without a window, a captured request is valid forever, and the window must absorb clock skew and retry delay.

Deduplicate by event id

Every provider sends an event identifier. Insert it into a table with a unique constraint and treat a conflict as already processed. This is the only reliable defence against duplicate delivery: retries, network duplication and your own redeploys all produce them.

Return 2xx fast, then process

Do the minimum work in the handler: verify, deduplicate, enqueue, return 200. Everything else runs in a worker. If you process synchronously and exceed the provider’s delivery timeout — commonly 5 to 10 seconds — the provider marks delivery failed and retries, so you do the work twice while also being slow.

What happens when your endpoint is down

Providers retry with exponential backoff for hours to a few days, then give up — a few fast retries, widening intervals, then silence. Your endpoint must be idempotent, because it will see the same event at one second and again at six hours. Webhooks also cannot be your only source of truth.

Treat webhooks as the fast path and a periodic reconciliation sweep as the slow path. The sweep queries the provider for anything running long and settles it locally. The webhook makes the system fast; the sweep makes it correct.

Idempotency for long jobs

Long jobs are expensive, which makes duplicate execution expensive. Two layers of protection are needed.

Idempotency keys on submission

Let the client supply an idempotency key on the POST, stored under a unique constraint scoped to the tenant, so a retried submission returns the original job id instead of creating a second job. The POST will be retried whether you plan for it or not: by the client’s HTTP library, by a load balancer, or by a double-click.

Bind the key to a hash of the request body. If the same key arrives with a different body, return 409 Conflict rather than silently returning the first job’s result. Otherwise a client reusing a constant key gets plausible-looking answers to questions it never asked.

Provider idempotency support varies

Some providers accept an Idempotency-Key header on some endpoints. Others document none, or scope it to a short window or a subset of routes. You cannot depend on the provider to deduplicate your retries, so design as though every retry may produce a second generation and a second charge.

Make your own side effects safe

Every effect your worker performs needs a key that makes repeating it harmless.

Result writes: INSERT ... ON CONFLICT (job_id) DO NOTHING.
Billing: a ledger entry keyed by (job_id, 'completion') under a unique constraint, so a redelivered completion cannot double-charge. Same discipline as usage metering and billing.
Notifications: deduplicate on (job_id, channel) before sending.
Downstream calls: propagate the job id as the downstream idempotency key.

Where an effect cannot be made idempotent, record the provider’s request identifier as soon as you receive it and, on retry, query that job’s status instead of resubmitting. That converts a duplicate generation into a cheap status lookup.

Polling done right

Polling is not the inferior pattern. It is correct when you have no public endpoint, low volume, or a need for a single authoritative state store — it just has to be done with backoff.

Exponential backoff with jitter

Fixed-interval polling wastes requests early and is too slow late. Exponential backoff with full jitter — sleeping a uniform random amount between zero and min(cap, base * 2^attempt) — spreads load and cuts request volume by roughly an order of magnitude. Without jitter, clients that started together stay synchronised and reproduce identical thundering-herd bursts.

Honour Retry-After

When a status endpoint returns 429 or 503 with a Retry-After header, that value overrides your backoff. Ignoring it is how well-behaved clients get throttled into uselessness, and it causes the 429 storms that look like provider outages but are self-inflicted.

Long polling

Long polling holds the connection open until the result is ready or a server-side timeout of 30 to 60 seconds elapses, collapsing many status requests into one. Most LLM providers do not offer it, so in practice it applies at your own gateway.

The cost of polling at scale

Assume 50,000 jobs per day with an average completion time of 4 minutes.

Naive polling every 2 seconds: 240 divided by 2 gives 120 polls per job, so 6,000,000 status requests per day. Spread over 86,400 seconds that is about 69 requests per second — and since submissions cluster in business hours, the peak is several times that.

Apply backoff starting at 1 second, doubling, capped at 30 seconds. Cumulative poll times are 1, 3, 7, 15, 31, 61, 91, 121, 151, 181, 211 and 241 seconds — 12 polls to cover a 240-second job, roughly a 10x reduction. The same jobs now generate 600,000 requests per day, about 7 per second.

The architectural point is where those requests land. Clients should poll your job table, never the provider directly. Your status endpoint is one indexed primary-key read; a provider status call consumes rate-limit budget and may cost money. Keep one background reconciler as the only component that talks to the provider.

Progress reporting and cancellation

Progress reporting should be coarse and honest. Expose the state, an attempt count, and a percentage only if the provider actually reports one. A synthesised percentage that creeps toward 90 percent and then stalls is worse than a plain “running” label.

Cancellation support varies by provider and endpoint. Some accept a cancel call that terminates a running job. Some only cancel jobs that have not started. Some have none at all, and the generation completes and bills regardless. Establish which before putting a cancel button in a UI.

When a job cannot be cancelled, fall back to compensating actions: mark it cancelled locally so the result is discarded on arrival, stop downstream work, and refund if you billed at submission. Make sure the failover layer knows too, or a cancelled job gets failed over and billed twice.

A worker that claims a job with a lease

This worker uses Postgres as the queue. It claims one job atomically, records the lease owner, calls the model outside any transaction, and writes the result only if it still holds the lease — which stops a reclaimed job from being written twice.

import os
import time
import uuid
import psycopg
from openai import OpenAI

LEASE_SECONDS = 60
MAX_ATTEMPTS = 3

client = OpenAI(
    api_key=os.environ["QORA_API_KEY"],
    base_url="https://api.qoraapi.com/v1",
)

CLAIM_SQL = """
UPDATE jobs
   SET state = 'running',
       attempt = attempt + 1,
       lease_owner = %(worker)s,
       lease_expires_at = now() + make_interval(secs => %(lease)s),
       started_at = COALESCE(started_at, now())
 WHERE id = (
     SELECT id FROM jobs
      WHERE state = 'queued'
         OR (state = 'running' AND lease_expires_at < now())
      ORDER BY created_at
      FOR UPDATE SKIP LOCKED
      LIMIT 1)
RETURNING id, attempt, payload;
"""

FINISH_SQL = """
UPDATE jobs
   SET state = %(state)s,
       result = %(result)s,
       error = %(error)s,
       lease_owner = NULL,
       lease_expires_at = NULL,
       finished_at = now()
 WHERE id = %(id)s
   AND lease_owner = %(worker)s
RETURNING id;
"""

def claim(conn, worker):
    with conn.cursor() as cur:
        cur.execute(CLAIM_SQL, {"worker": worker, "lease": LEASE_SECONDS})
        return cur.fetchone()

def finish(conn, worker, job_id, state, result=None, error=None):
    with conn.cursor() as cur:
        cur.execute(FINISH_SQL, {
            "id": job_id, "worker": worker,
            "state": state, "result": result, "error": error,
        })
        # No row returned means the lease was reclaimed; discard our result.
        return cur.fetchone() is not None

def run(worker):
    with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
        conn.autocommit = True
        while True:
            job = claim(conn, worker)
            if job is None:
                time.sleep(1)
                continue
            job_id, attempt, payload = job
            try:
                resp = client.chat.completions.create(
                    model=payload["model"],
                    messages=payload["messages"],
                    timeout=600.0,
                )
                text = resp.choices[0].message.content
                finish(conn, worker, job_id, "succeeded", result=text)
            except Exception as exc:
                state = "failed" if attempt >= MAX_ATTEMPTS else "queued"
                finish(conn, worker, job_id, state, error=str(exc))

if __name__ == "__main__":
    run(f"worker-{uuid.uuid4()}")

The same lease concept as SQL, which is also what a reaper runs on a timer:

-- Reclaim jobs whose worker stopped heartbeating, while attempts remain.
UPDATE jobs
   SET state = 'queued',
       lease_owner = NULL,
       lease_expires_at = NULL
 WHERE state = 'running'
   AND lease_expires_at < now()
   AND attempt < 3;

-- Give up on jobs that exhausted their attempts.
UPDATE jobs
   SET state = 'failed',
       error = COALESCE(error, 'lease expired after max attempts')
 WHERE state = 'running'
   AND lease_expires_at < now()
   AND attempt >= 3;

Webhook verification in practice

The handler below is deliberately boring: verify, deduplicate, enqueue, acknowledge. The unusual detail is that it works on the raw request buffer, which is what makes the signature check correct.

import crypto from "node:crypto";
import type { Request, Response } from "express";
import { pool } from "./db";
import { enqueue } from "./queue";

const TOLERANCE_SECONDS = 300;

function verify(rawBody: Buffer, header: string, secret: string): boolean {
  const parts = Object.fromEntries(
    header.split(",").map((kv) => kv.split("=") as [string, string]),
  );

  const timestamp = Number(parts.t);
  if (!Number.isFinite(timestamp)) return false;
  if (Math.abs(Date.now() / 1000 - timestamp) > TOLERANCE_SECONDS) return false;

  const expected = crypto
    .createHmac("sha256", secret)
    .update(`${parts.t}.`)
    .update(rawBody)
    .digest("hex");

  const a = Buffer.from(expected);
  const b = Buffer.from(parts.v1 ?? "");
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

// Mount with express.raw({ type: "application/json" }) so req.body is a Buffer.
export async function webhookHandler(req: Request, res: Response) {
  const raw = req.body as Buffer;
  const header = req.header("x-provider-signature") ?? "";

  if (!verify(raw, header, process.env.WEBHOOK_SECRET ?? "")) {
    res.status(401).end();
    return;
  }

  const event = JSON.parse(raw.toString("utf8"));

  const inserted = await pool.query(
    `INSERT INTO webhook_events (event_id, received_at)
     VALUES ($1, now()) ON CONFLICT (event_id) DO NOTHING
     RETURNING event_id`,
    [event.id],
  );

  // Duplicate delivery: acknowledge without reprocessing.
  if (inserted.rowCount === 0) {
    res.status(200).end();
    return;
  }

  await enqueue("job-results", { eventId: event.id, payload: event.data });
  res.status(200).end();
}

Operational concerns

The stuck-job reaper

Run the reaper every 30 to 60 seconds and make it idempotent, so a double run is harmless. Alert when one pass reclaims more than a small threshold: a burst means workers are crashing in bulk, an infrastructure problem rather than bad luck.

Alert on queue age, not just depth

Queue depth alone is misleading. Ten thousand pending jobs is comfortable at 50 milliseconds each and catastrophic at five minutes each. Alert on the age of the oldest unclaimed message and the p95 time from queued to running.

Per-tenant fairness

One tenant submitting 100,000 jobs must not starve everyone else. The cheapest control is a per-tenant concurrency cap enforced at claim time, so no tenant holds more than N worker slots. Weighted fair queueing is the fuller answer: order candidates by how far each tenant sits below its fair share.

Backpressure when the provider is slow

When the provider returns 429s or latency climbs, the instinct is to retry harder. That is backwards. Slow the consumers, honour Retry-After, and trip a circuit breaker after consecutive failures. Bound the queue and reject new submissions with 429 once full: refusing a job immediately beats accepting one you cannot finish.

When webhooks are worth it

My default is submit-and-poll with a reconciliation worker, and webhooks only once volume or latency makes polling genuinely expensive. Polling has no failure mode you have not already handled: your status endpoint is the source of truth, and a client that goes away leaves no dangling state.

Webhooks invert that trade. You gain delivery latency measured in milliseconds instead of your poll interval, and you pay with a public endpoint, HMAC verification, a replay window, deduplication, retry handling, and a reconciliation sweep anyway.

They are worth it when the job is user-facing and long, when you deliver tens of thousands of completions per day, or when the consumer is a server you control. A gateway that normalises providers behind one endpoint lets you start with polling and add webhooks later without changing your client contract. Qora API is one option: one OpenAI-compatible key across GPT, Claude, Gemini and others, so your delivery logic survives a provider swap.

Frequently asked questions

How fast should a webhook endpoint respond?

Under five seconds, and ideally under 500 milliseconds. Verify the signature, insert the event id under a unique constraint, enqueue, return 200. Anything else belongs in a worker. A handler that occasionally takes 8 seconds against a 10-second delivery timeout accumulates retries during spikes, precisely when you can least afford them.

Can I get exactly-once delivery?

No, not end to end. At-least-once transport plus idempotent consumers is the achievable target, and it yields exactly-once effects as long as every side effect is keyed. Treat any claim of exactly-once delivery spanning a queue, a database and a third-party API as a description of deduplication.

What if a provider never sends the webhook?

You must detect it. Record the expected delivery deadline at submission and run a sweep for anything past it. If the provider reports the job finished, settle it locally and emit the event yourself. Without the sweep, a webhook lost during a deploy becomes a job stuck forever.

Should I use streaming or webhooks?

They solve different problems. Streaming optimises time-to-first-token for a user watching output appear; webhooks optimise completion notification for a job nobody is watching. A long document analysis might stream partial sections and still fire a webhook when the structured result is ready.

How do I choose a lease length?

Start from your p99 job duration and add margin, then add a heartbeat so the value need not cover worst-case runtimes. A 60-second lease with a 15-second heartbeat handles jobs of any length while keeping reclaim latency under a minute. Sizing the timeout to the worst case means a crashed worker blocks that job for its full duration.

Conclusion

Asynchronous delivery is not a feature you bolt onto an API; it is a different contract. Once submission and completion are separated you own a durable job row, a lease, a retry policy, an idempotency story and a reconciliation loop. In return, a 90-second generation stops being a failure and becomes ordinary.

Build the state machine first, get the lease and idempotency keys right, and start with polling. Add webhooks when you can point at a specific latency or cost problem that polling causes. Keep the reconciliation sweep running either way, because it turns an at-least-once world into a correct one. A unified gateway such as Qora API reduces that provider-specific surface to one endpoint and one billing model.

Async AI APIs: Job Queues, Webhooks, and Long-Running Tasks