Batch AI APIs: Processing Millions of Requests Affordably

A batch AI API lets you submit thousands of requests as one asynchronous job, then collect the results when the job finishes. You trade minutes-to-hours of latency for a large per-token discount and no concurrency management. It is the right tool for offline classification, embedding backfills, enrichment, and evals — never for anything a user waits on.

This guide covers the production batch pipeline end to end: what belongs in batch at all, the submit-poll-collect loop with working code, idempotent chunking, surgical retries, and throughput sizing.

When batch beats realtime

Three tests decide it. If all three pass, batch is almost always the correct choice:

• No human is blocked. Nothing in the product is holding a spinner or a connection open waiting for this result.
• The work is embarrassingly parallel. Each item’s prompt is self-contained. If item N‘s prompt needs item N-1‘s output, batch is structurally wrong — that is an agent loop, and it needs realtime calls.
• The result is still useful when it is hours late. A category label that lands six hours from now is fine for a catalog. A fraud score that lands six hours from now is worthless.

The third test is the one teams get wrong. Staleness tolerance is a product decision, not an engineering one, so make it explicit before you write the job:

Workload	Batch or realtime	Why
Catalog classification across millions of SKUs	Batch	Labels are refreshed on a schedule; nothing reads them synchronously
Embedding backfill for a new index	Batch	Write-once corpus — the read path does not exist until the index is built
CRM / company enrichment	Batch	Hour-scale staleness is invisible to the user of the enriched record
Offline evals and regression suites	Batch	Latency is irrelevant; cost per run and reproducibility are everything
Backlog moderation pre-screen	Batch	The queue is already asynchronous; only the flagged subset needs a human
Nightly summarization of the day’s tickets	Batch	Hard deadline hours away, so the completion window is a contract you can meet
Live chat assistant	Realtime	A user is watching tokens appear
Inline autocomplete	Realtime	Sub-second budget; batch turnaround is measured in hours
Agent tool-calling loops	Realtime	Each turn depends on the previous turn’s output
Checkout fraud scoring	Realtime	The score gates a transaction that is happening right now

One hybrid pattern is worth knowing: split the job, not the pipeline. A nightly report might use batch for the expensive extraction over 400,000 rows, then a single realtime call to compose the executive summary once the batch results land. You get the discount on 99.9% of the tokens and keep the interactive step fast.

How batch APIs work

Every major provider’s batch interface follows the same four-phase shape, which is why the code below ports between them with one changed base URL:

• Build a JSONL file: one line per request, each carrying a custom_id, a method, a url, and a body identical to what you would POST to the realtime endpoint.
• Upload the file to the provider’s file store and reference its id when creating the job.
• Poll the job. The only progress signal is a request-count object with completed / failed / total.
• Collect the output file and join it back to your data by custom_id.

That last step is the first production trap: output order is not guaranteed to match input order, and it is not guaranteed to be complete. Join on custom_id, never on line number. Here is a working submit-poll-collect loop:

import json, os, time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["QORA_API_KEY"],
    base_url="https://api.qoraapi.com/v1",   # one key, many models
)

def build_jsonl(items, path, model="gpt-4o-mini"):
    """One JSON object per line: custom_id + the body you'd send to /chat/completions."""
    with open(path, "w", encoding="utf-8") as f:
        for it in items:
            f.write(json.dumps({
                "custom_id": f"sku-{it['id']}",        # stable and deterministic
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": model,
                    "messages": [
                        {"role": "system", "content": "Return JSON: {\"category\": str, \"confidence\": float}"},
                        {"role": "user", "content": it["text"]},
                    ],
                    "response_format": {"type": "json_object"},
                    "temperature": 0,                   # reproducibility for evals
                },
            }) + "\n")

def submit(path):
    upload = client.files.create(file=open(path, "rb"), purpose="batch")
    job = client.batches.create(
        input_file_id=upload.id,
        endpoint="/v1/chat/completions",
        completion_window="24h",
        metadata={"pipeline": "sku-classify", "run": os.environ["RUN_ID"]},
    )
    return job.id

def poll(job_id, every=30, timeout=6 * 3600):
    deadline = time.time() + timeout
    while time.time() < deadline:
        job = client.batches.retrieve(job_id)
        counts = job.request_counts
        print(f"{job.status}: {counts.completed}/{counts.total} failed={counts.failed}")
        if job.status in ("completed", "failed", "cancelled", "expired"):
            return job
        time.sleep(every)
    raise TimeoutError(f"job {job_id} still running after {timeout}s")

def collect(job, out_path):
    """Join results back by custom_id. Never assume input order."""
    results = {}
    if job.output_file_id:
        for line in client.files.content(job.output_file_id).text.splitlines():
            row = json.loads(line)
            body = row["response"]["body"]
            results[row["custom_id"]] = (
                json.loads(body["choices"][0]["message"]["content"])
                if row["response"]["status_code"] == 200
                else {"error": body}
            )
    with open(out_path, "w", encoding="utf-8") as f:
        json.dump(results, f)
    return results

if __name__ == "__main__":
    build_jsonl(load_skus(), "input.jsonl")
    job = poll(submit("input.jsonl"))
    results = collect(job, "results.json")
    print(f"{len(results)} results, error_file={job.error_file_id}")

Two fields matter more than the rest. request_counts is your only progress signal — poll it, do not infer progress from file sizes. And error_file_id is a separate artifact from the output file: it holds lines that failed at the request level (malformed JSON, oversized input, expired window), while the output file holds per-line status codes for everything the model actually saw. You need both to reconcile a run.

Cost and latency trade-off

Batch pricing is a pricing tier, not a quality tier. The same model weights answer your requests; the discount exists because you gave up the ability to demand a response now. In practice the batch rate lands at roughly 0.4×–0.6× the realtime rate for the same model, and some providers stack a cached-input discount on top for shared prefixes. Because the ratio is far more stable than any absolute price, budget in ratios:

# Cost model that survives price changes: work in ratios, not dollar amounts.
run_cost_realtime = items * avg_tokens * realtime_rate
run_cost_batch    = items * avg_tokens * realtime_rate * batch_ratio   # batch_ratio ~ 0.4-0.6

# The discount you actually bank is larger than batch_ratio suggests, because a
# realtime fan-out also pays for the retries it causes:
realtime_overhead = 1 + (rate_limit_error_rate * retry_multiplier)   # 429 retries, idle workers
effective_saving  = 1 - (batch_ratio / realtime_overhead)

# Retry cost, charged at the batch rate, is the third term:
retry_cost = items * error_rate * retry_rate * attempts

The second term is the part most cost models miss. A realtime fan-out across 200 workers will hit rate limits, and every 429 you retry is a token you paid for twice. Batch eliminates that entire class of waste because the provider owns the queue — which is why a measured batch migration often beats the headline discount. For the other levers that stack with batch (prompt caching, token budgeting, tier routing), see our guide to reduce AI API costs.

Latency, meanwhile, is a contract you choose. Providers expose different completion windows, and the window you request changes both the discount and your tolerance for queue depth:

Turnaround you request	What it buys	Realistic p50 in practice	Fits
24-hour window	Deepest discount, tolerates a congested queue	Tens of minutes to a few hours	Nightly jobs, multi-million-item backfills
Same-day / 12-hour window	Middle ground — smaller discount, tighter queue	1–4 hours	Intraday refresh, enrichment on a business-day SLA
No batch (realtime)	Lowest latency, full price, you own concurrency	Sub-second to seconds	Anything a user or an agent loop is waiting on

Design against the window, not the p50. If the pipeline must be complete by 06:00, submit at 22:00 with a 24-hour window and treat 24 hours as the worst case you are willing to absorb. That single habit turns "the batch was slow last night" from an incident into a scheduling parameter.

Designing idempotent batch jobs

A batch job that cannot be safely re-run is a batch job you will eventually re-run by accident. Four design choices make the pipeline idempotent:

• Chunk deterministically. Sort by item_id and slice, or bucket by hash(item_id) % N. Never chunk by "whatever arrived in this batch" — if chunk membership drifts between runs, you reprocess items you already paid for.
• Derive the job id from content. job_id = f"{pipeline}:{prompt_hash}:{chunk_index}". Hashing the normalized prompt bodies means a prompt edit produces a new job id (correct — the old results are stale), while a re-run of an unchanged chunk collides and is skipped.
• Persist state in a table, not in the process. Record job_id → submitted | running | collected | failed plus the provider's job id. The driver reads that table and submits only chunks with no terminal row. A crashed worker then resumes by reading state, not by guessing.
• Dedupe before you submit. Hash the normalized prompt and collapse identical items into one custom_id, then fan the single result back out to every source row. In classification and enrichment corpora, 10–30% duplicate rates are normal, and those are free wins at the batch rate.

import hashlib, json

def chunk_key(pipeline, item_id, n_chunks):
    """Stable across runs: same item always lands in the same chunk."""
    h = hashlib.sha256(f"{pipeline}:{item_id}".encode()).hexdigest()
    return int(h[:8], 16) % n_chunks

def prompt_hash(bodies):
    """Hash normalized bodies so a prompt edit invalidates old results."""
    norm = json.dumps(bodies, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(norm.encode()).hexdigest()[:16]

# job_id is a function of (pipeline, prompt, chunk) — so re-running a chunk
# with an unchanged prompt produces the SAME id and is skipped by the driver.
job_id = f"{pipeline}:{prompt_hash(chunk_bodies)}:{chunk_index}"

The subtle win here is that idempotency and resume become the same mechanism. Because the job id encodes the prompt version, a re-run after a partial failure cannot silently mix old and new prompt results in one table — the old rows keep the old hash, and you can see exactly which items were produced by which prompt revision. That is also what makes offline evals trustworthy: the eval set and the production run use the same chunking code, so a score difference is a prompt difference, not a data difference.

Handling partial failures and retries

Batch jobs fail by the item, not by the job. A 1-million-item run that returns 98.5% success is a good run — but only if you handle the 15,000 failures surgically. The output file gives you a per-line status code, and the error file gives you the request-level failures, so the first step is always to build a per-item error map:

errors = {}
for line in output_lines:
    row = json.loads(line)
    code = row["response"]["status_code"]
    if code != 200:
        errors[row["custom_id"]] = {"code": code, "body": row["response"]["body"], "attempts": 1}

# Classify before you retry — the classification decides the action.
RETRYABLE = {429, 500, 502, 503, 504}
for cid, err in errors.items():
    if err["code"] in RETRYABLE:
        retry_queue.append(cid)          # transient: safe to resubmit
    elif err["code"] in (400, 422):
        dead_letter.append(cid)          # schema or prompt bug: retrying burns money
    else:
        dead_letter.append(cid)          # investigate, don't loop

Three rules keep this from becoming an accidental second full run:

• Never resubmit the whole job. At a 2% error rate, resubmitting all 1M items spends 98% of a fresh run's budget re-buying results you already have. A per-item retry costs roughly 1 + error_rate × attempts of the base run; whole-job resubmission costs a full multiple per round.
• Do not retry deterministic errors. A 400 or 422 means the request body or the schema is wrong. The same input will produce the same error every time. Fix the prompt, then submit those items as a new job with a new prompt hash — and route them through your dead-letter table so they are visible.
• Cap attempts and record terminal failures. Three attempts is enough for transient errors. After that, write the item to a failed table with its last error so downstream consumers can decide between a fallback model and a human review.

Retries are billable, so they belong in your cost accounting rather than a log line nobody reads. If you are attributing spend per tenant, per pipeline, or per customer, the retry rounds have to land in the same ledger as the base run — which is exactly what metering AI usage is for. Without it, a pipeline with a flaky 5% error rate looks 5% more expensive than it is, and nobody notices until the invoice arrives.

Throughput planning

Batch removes concurrency management from your side but not from the provider's. Your ceiling is now a small set of hard limits: max requests per input file, max concurrent jobs per account, max items in flight, and the requests-per-minute cap on the realtime submit and poll calls themselves.

The planning math is one equation. If you must process N items per day, each job carries C items, and turnaround is T hours, the number of jobs you need in flight simultaneously is:

concurrent_jobs = (N * T / 24) / C

# 10M items/day, 50k items per job, 6h turnaround:
# (10_000_000 * 6 / 24) / 50_000 = 50 concurrent jobs
#
# If the account cap is 20 concurrent jobs, your ceiling with this chunk size is:
# 20 * 50_000 * 24 / 6 = 4M items/day  ->  you are 2.5x short of the target.
#
# Fixes, in order of least pain:
#   1. Shrink T (request a faster window, if one is offered)
#   2. Raise the cap with the provider
#   3. Shard across two accounts/providers via a gateway

Note what that equation implies: chunk size is a throughput lever, not just a failure-domain lever. Larger chunks mean fewer concurrent jobs for the same volume, but a single failure costs more items. The usual sweet spot is the largest chunk your retry budget can tolerate — if re-running one chunk is acceptable at your error rate, the chunk is not too big.

Do not forget the control plane. Submitting 50 jobs and polling each every 10 seconds is 5 requests per second of pure status traffic, and those calls hit the same per-key rate limits as your production traffic. Poll with exponential backoff (start at 30s, cap at 5 minutes for long windows), and expect the occasional 429 on a status check — the handling is the same as any other throttled call, covered in our guide to rate limits and 429 errors. A job that is throttled on polling has not failed; it is just being checked too eagerly.

Lever	Effect on throughput	Cost of pulling it
Larger chunk size	Fewer concurrent jobs needed	Bigger blast radius per failure
Shorter completion window	Lower T, fewer jobs in flight	Smaller discount
More concurrent jobs	Linear gain, up to the account cap	Requires a provider-side raise
Dedupe before submit	Cuts N directly	None — pure win
Multi-provider sharding	Multiplies the cap	Two integrations, unless you use a gateway

Running batch through a gateway

Every provider implements batch slightly differently — different upload endpoints, different job status enums, different output schemas, different window names. Supporting three of them means three sets of submit/poll/collect code and three places for a bug to hide.

An OpenAI-compatible gateway collapses that. The same build_jsonl / submit / poll / collect functions from earlier run unchanged; you move between models by editing the model string inside each line's body. That makes two patterns practical that are painful otherwise:

• Per-item model routing inside one run. Cheap items go to a small/fast model, ambiguous items to a mid model — same file, same job, one polling loop. Split into per-model chunks only when you need per-model SLAs.
• Uniform usage records. One invoice and one usage record per custom_id means per-tenant chargeback and retry accounting come from a single source instead of three dashboards.

That is the specific problem an AI API relay solves for batch workloads: one key, many models, and a single job-submission shape in front of all of them. qoraapi.com exposes many models behind one OpenAI-compatible endpoint, so the pipeline above does not need a provider-specific branch when you add a model or absorb an outage.

Frequently asked questions

Is batch cheaper than realtime for the same model?

Yes — and it is the same model, so there is no quality penalty. Batch is a pricing tier for asynchronous delivery, typically landing around 0.4×–0.6× the realtime rate. The measured saving is often larger, because batch also eliminates the retry waste a realtime fan-out generates when it hits rate limits.

How long does a batch job take?

You choose a completion window (commonly 24 hours, sometimes shorter) and the provider commits to finishing inside it. Actual p50 is usually far faster — tens of minutes to a few hours for typical job sizes — but you should schedule against the window, not the median. Treat the window as the worst case your pipeline is designed to absorb.

Can I use batch for streaming or interactive features?

No. Batch returns results only after the job completes, so there is no partial token stream to forward and no way to answer a request that depends on the previous one. Interactive UX and agent loops need realtime calls; batch is for work where nothing is waiting.

What happens when a batch job expires?

Results for items that completed are still written to the output file, and the unfinished items appear as failures in the error file. Reconcile by custom_id, then resubmit only the missing ids — never the whole job. If expiry happens repeatedly, your chunk size is too large for the window you requested.

Conclusion

Batch is the highest-leverage cost lever available to an offline AI pipeline, and it is not a drop-in switch — it is an architecture. Decide with the three tests, then build so the pipeline can be re-run safely: deterministic chunks, content-derived job ids, state in a table, dedupe before you submit. Retry per item, never per job, and classify errors before you spend money on them.

Finally, size the system with the throughput equation before you launch, not after the first job misses its deadline. If you want the batch lifecycle in front of many models without writing a provider branch for each one, start from the AI API gateway guide and the OpenAI-compatible API explainer, then point the code above at a single endpoint.

Related reading

• How to Reduce AI API Costs: A Practical Guide for Developers
• How to Handle AI API Rate Limits and 429 Errors
• Metering and Billing AI Usage Per User: A Practical SaaS Guide
• Extracting Structured Data from Documents with AI APIs
• Local LLMs vs API: A Real Cost and Latency Comparison for 2026
• What Is the Model Context Protocol (MCP)? Connect Your AI to Real Tools
• How to Connect Cursor, Cline and Continue to a Custom AI API Endpoint