{"id":136,"date":"2026-09-17T15:47:47","date_gmt":"2026-09-17T07:47:47","guid":{"rendered":"https:\/\/wp.qoraapi.com\/batch-ai-api-processing\/"},"modified":"2026-09-20T03:53:34","modified_gmt":"2026-09-19T19:53:34","slug":"batch-ai-api-processing","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/batch-ai-api-processing\/","title":{"rendered":"Batch AI APIs: Processing Millions of Requests Affordably"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A batch AI API lets you submit thousands of requests as one asynchronous job, then collect the results when the job finishes. You trade minutes-to-hours of latency for a large per-token discount and no concurrency management. It is the right tool for offline classification, embedding backfills, enrichment, and evals \u2014 never for anything a user waits on.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This guide covers the production batch pipeline end to end: what belongs in batch at all, the submit-poll-collect loop with working code, idempotent chunking, surgical retries, and throughput sizing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">When batch beats realtime<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Three tests decide it. If all three pass, batch is almost always the correct choice:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>No human is blocked.<\/strong> Nothing in the product is holding a spinner or a connection open waiting for this result.<\/li>\n<li><strong>The work is embarrassingly parallel.<\/strong> Each item&#8217;s prompt is self-contained. If item <em>N<\/em>&#8216;s prompt needs item <em>N-1<\/em>&#8216;s output, batch is structurally wrong \u2014 that is an agent loop, and it needs realtime calls.<\/li>\n<li><strong>The result is still useful when it is hours late.<\/strong> A category label that lands six hours from now is fine for a catalog. A fraud score that lands six hours from now is worthless.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The third test is the one teams get wrong. Staleness tolerance is a product decision, not an engineering one, so make it explicit before you write the job:<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Workload<\/th><th>Batch or realtime<\/th><th>Why<\/th><\/tr><\/thead><tbody><tr><td>Catalog classification across millions of SKUs<\/td><td>Batch<\/td><td>Labels are refreshed on a schedule; nothing reads them synchronously<\/td><\/tr><tr><td>Embedding backfill for a new index<\/td><td>Batch<\/td><td>Write-once corpus \u2014 the read path does not exist until the index is built<\/td><\/tr><tr><td>CRM \/ company enrichment<\/td><td>Batch<\/td><td>Hour-scale staleness is invisible to the user of the enriched record<\/td><\/tr><tr><td>Offline evals and regression suites<\/td><td>Batch<\/td><td>Latency is irrelevant; cost per run and reproducibility are everything<\/td><\/tr><tr><td>Backlog moderation pre-screen<\/td><td>Batch<\/td><td>The queue is already asynchronous; only the flagged subset needs a human<\/td><\/tr><tr><td>Nightly summarization of the day&#8217;s tickets<\/td><td>Batch<\/td><td>Hard deadline hours away, so the completion window is a contract you can meet<\/td><\/tr><tr><td>Live chat assistant<\/td><td>Realtime<\/td><td>A user is watching tokens appear<\/td><\/tr><tr><td>Inline autocomplete<\/td><td>Realtime<\/td><td>Sub-second budget; batch turnaround is measured in hours<\/td><\/tr><tr><td>Agent tool-calling loops<\/td><td>Realtime<\/td><td>Each turn depends on the previous turn&#8217;s output<\/td><\/tr><tr><td>Checkout fraud scoring<\/td><td>Realtime<\/td><td>The score gates a transaction that is happening right now<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">One hybrid pattern is worth knowing: <strong>split the job, not the pipeline<\/strong>. A nightly report might use batch for the expensive extraction over 400,000 rows, then a single realtime call to compose the executive summary once the batch results land. You get the discount on 99.9% of the tokens and keep the interactive step fast.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How batch APIs work<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Every major provider&#8217;s batch interface follows the same four-phase shape, which is why the code below ports between them with one changed base URL:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Build<\/strong> a JSONL file: one line per request, each carrying a <code>custom_id<\/code>, a <code>method<\/code>, a <code>url<\/code>, and a <code>body<\/code> identical to what you would POST to the realtime endpoint.<\/li>\n<li><strong>Upload<\/strong> the file to the provider&#8217;s file store and reference its id when creating the job.<\/li>\n<li><strong>Poll<\/strong> the job. The only progress signal is a request-count object with completed \/ failed \/ total.<\/li>\n<li><strong>Collect<\/strong> the output file and join it back to your data by <code>custom_id<\/code>.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">That last step is the first production trap: <strong>output order is not guaranteed to match input order<\/strong>, and it is not guaranteed to be complete. Join on <code>custom_id<\/code>, never on line number. Here is a working submit-poll-collect loop:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import json, os, time\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=os.environ[\"QORA_API_KEY\"],\n    base_url=\"https:\/\/api.qoraapi.com\/v1\",   # one key, many models\n)\n\ndef build_jsonl(items, path, model=\"gpt-4o-mini\"):\n    \"\"\"One JSON object per line: custom_id + the body you'd send to \/chat\/completions.\"\"\"\n    with open(path, \"w\", encoding=\"utf-8\") as f:\n        for it in items:\n            f.write(json.dumps({\n                \"custom_id\": f\"sku-{it['id']}\",        # stable and deterministic\n                \"method\": \"POST\",\n                \"url\": \"\/v1\/chat\/completions\",\n                \"body\": {\n                    \"model\": model,\n                    \"messages\": [\n                        {\"role\": \"system\", \"content\": \"Return JSON: {\\\"category\\\": str, \\\"confidence\\\": float}\"},\n                        {\"role\": \"user\", \"content\": it[\"text\"]},\n                    ],\n                    \"response_format\": {\"type\": \"json_object\"},\n                    \"temperature\": 0,                   # reproducibility for evals\n                },\n            }) + \"\\n\")\n\ndef submit(path):\n    upload = client.files.create(file=open(path, \"rb\"), purpose=\"batch\")\n    job = client.batches.create(\n        input_file_id=upload.id,\n        endpoint=\"\/v1\/chat\/completions\",\n        completion_window=\"24h\",\n        metadata={\"pipeline\": \"sku-classify\", \"run\": os.environ[\"RUN_ID\"]},\n    )\n    return job.id\n\ndef poll(job_id, every=30, timeout=6 * 3600):\n    deadline = time.time() + timeout\n    while time.time() < deadline:\n        job = client.batches.retrieve(job_id)\n        counts = job.request_counts\n        print(f\"{job.status}: {counts.completed}\/{counts.total} failed={counts.failed}\")\n        if job.status in (\"completed\", \"failed\", \"cancelled\", \"expired\"):\n            return job\n        time.sleep(every)\n    raise TimeoutError(f\"job {job_id} still running after {timeout}s\")\n\ndef collect(job, out_path):\n    \"\"\"Join results back by custom_id. Never assume input order.\"\"\"\n    results = {}\n    if job.output_file_id:\n        for line in client.files.content(job.output_file_id).text.splitlines():\n            row = json.loads(line)\n            body = row[\"response\"][\"body\"]\n            results[row[\"custom_id\"]] = (\n                json.loads(body[\"choices\"][0][\"message\"][\"content\"])\n                if row[\"response\"][\"status_code\"] == 200\n                else {\"error\": body}\n            )\n    with open(out_path, \"w\", encoding=\"utf-8\") as f:\n        json.dump(results, f)\n    return results\n\nif __name__ == \"__main__\":\n    build_jsonl(load_skus(), \"input.jsonl\")\n    job = poll(submit(\"input.jsonl\"))\n    results = collect(job, \"results.json\")\n    print(f\"{len(results)} results, error_file={job.error_file_id}\")\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two fields matter more than the rest. <code>request_counts<\/code> is your only progress signal \u2014 poll it, do not infer progress from file sizes. And <code>error_file_id<\/code> is a separate artifact from the output file: it holds lines that failed at the request level (malformed JSON, oversized input, expired window), while the output file holds per-line status codes for everything the model actually saw. You need both to reconcile a run.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Cost and latency trade-off<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Batch pricing is a <strong>pricing tier, not a quality tier<\/strong>. The same model weights answer your requests; the discount exists because you gave up the ability to demand a response now. In practice the batch rate lands at roughly <strong>0.4\u00d7\u20130.6\u00d7 the realtime rate<\/strong> for the same model, and some providers stack a cached-input discount on top for shared prefixes. Because the ratio is far more stable than any absolute price, budget in ratios:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Cost model that survives price changes: work in ratios, not dollar amounts.\nrun_cost_realtime = items * avg_tokens * realtime_rate\nrun_cost_batch    = items * avg_tokens * realtime_rate * batch_ratio   # batch_ratio ~ 0.4-0.6\n\n# The discount you actually bank is larger than batch_ratio suggests, because a\n# realtime fan-out also pays for the retries it causes:\nrealtime_overhead = 1 + (rate_limit_error_rate * retry_multiplier)   # 429 retries, idle workers\neffective_saving  = 1 - (batch_ratio \/ realtime_overhead)\n\n# Retry cost, charged at the batch rate, is the third term:\nretry_cost = items * error_rate * retry_rate * attempts\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The second term is the part most cost models miss. A realtime fan-out across 200 workers will hit rate limits, and every 429 you retry is a token you paid for twice. Batch eliminates that entire class of waste because the provider owns the queue \u2014 which is why a measured batch migration often beats the headline discount. For the other levers that stack with batch (prompt caching, token budgeting, tier routing), see our guide to <a href=\"https:\/\/qoraapi.com\/blog\/reduce-ai-api-costs\/\">reduce AI API costs<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Latency, meanwhile, is a contract you choose. Providers expose different completion windows, and the window you request changes both the discount and your tolerance for queue depth:<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Turnaround you request<\/th><th>What it buys<\/th><th>Realistic p50 in practice<\/th><th>Fits<\/th><\/tr><\/thead><tbody><tr><td>24-hour window<\/td><td>Deepest discount, tolerates a congested queue<\/td><td>Tens of minutes to a few hours<\/td><td>Nightly jobs, multi-million-item backfills<\/td><\/tr><tr><td>Same-day \/ 12-hour window<\/td><td>Middle ground \u2014 smaller discount, tighter queue<\/td><td>1\u20134 hours<\/td><td>Intraday refresh, enrichment on a business-day SLA<\/td><\/tr><tr><td>No batch (realtime)<\/td><td>Lowest latency, full price, you own concurrency<\/td><td>Sub-second to seconds<\/td><td>Anything a user or an agent loop is waiting on<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Design against the <em>window<\/em>, not the p50. If the pipeline must be complete by 06:00, submit at 22:00 with a 24-hour window and treat 24 hours as the worst case you are willing to absorb. That single habit turns \"the batch was slow last night\" from an incident into a scheduling parameter.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Designing idempotent batch jobs<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A batch job that cannot be safely re-run is a batch job you will eventually re-run by accident. Four design choices make the pipeline idempotent:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Chunk deterministically.<\/strong> Sort by <code>item_id<\/code> and slice, or bucket by <code>hash(item_id) % N<\/code>. Never chunk by \"whatever arrived in this batch\" \u2014 if chunk membership drifts between runs, you reprocess items you already paid for.<\/li>\n<li><strong>Derive the job id from content.<\/strong> <code>job_id = f\"{pipeline}:{prompt_hash}:{chunk_index}\"<\/code>. Hashing the normalized prompt bodies means a prompt edit produces a new job id (correct \u2014 the old results are stale), while a re-run of an unchanged chunk collides and is skipped.<\/li>\n<li><strong>Persist state in a table, not in the process.<\/strong> Record <code>job_id \u2192 submitted | running | collected | failed<\/code> plus the provider's job id. The driver reads that table and submits only chunks with no terminal row. A crashed worker then resumes by reading state, not by guessing.<\/li>\n<li><strong>Dedupe before you submit.<\/strong> Hash the normalized prompt and collapse identical items into one <code>custom_id<\/code>, then fan the single result back out to every source row. In classification and enrichment corpora, 10\u201330% duplicate rates are normal, and those are free wins at the batch rate.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>import hashlib, json\n\ndef chunk_key(pipeline, item_id, n_chunks):\n    \"\"\"Stable across runs: same item always lands in the same chunk.\"\"\"\n    h = hashlib.sha256(f\"{pipeline}:{item_id}\".encode()).hexdigest()\n    return int(h[:8], 16) % n_chunks\n\ndef prompt_hash(bodies):\n    \"\"\"Hash normalized bodies so a prompt edit invalidates old results.\"\"\"\n    norm = json.dumps(bodies, sort_keys=True, separators=(\",\", \":\"))\n    return hashlib.sha256(norm.encode()).hexdigest()[:16]\n\n# job_id is a function of (pipeline, prompt, chunk) \u2014 so re-running a chunk\n# with an unchanged prompt produces the SAME id and is skipped by the driver.\njob_id = f\"{pipeline}:{prompt_hash(chunk_bodies)}:{chunk_index}\"\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The subtle win here is that idempotency and resume become the same mechanism. Because the job id encodes the prompt version, a re-run after a partial failure cannot silently mix old and new prompt results in one table \u2014 the old rows keep the old hash, and you can see exactly which items were produced by which prompt revision. That is also what makes offline evals trustworthy: the eval set and the production run use the same chunking code, so a score difference is a prompt difference, not a data difference.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Handling partial failures and retries<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Batch jobs fail by the item, not by the job. A 1-million-item run that returns 98.5% success is a good run \u2014 but only if you handle the 15,000 failures surgically. The output file gives you a per-line status code, and the error file gives you the request-level failures, so the first step is always to build a per-item error map:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>errors = {}\nfor line in output_lines:\n    row = json.loads(line)\n    code = row[\"response\"][\"status_code\"]\n    if code != 200:\n        errors[row[\"custom_id\"]] = {\"code\": code, \"body\": row[\"response\"][\"body\"], \"attempts\": 1}\n\n# Classify before you retry \u2014 the classification decides the action.\nRETRYABLE = {429, 500, 502, 503, 504}\nfor cid, err in errors.items():\n    if err[\"code\"] in RETRYABLE:\n        retry_queue.append(cid)          # transient: safe to resubmit\n    elif err[\"code\"] in (400, 422):\n        dead_letter.append(cid)          # schema or prompt bug: retrying burns money\n    else:\n        dead_letter.append(cid)          # investigate, don't loop\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Three rules keep this from becoming an accidental second full run:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Never resubmit the whole job.<\/strong> At a 2% error rate, resubmitting all 1M items spends 98% of a fresh run's budget re-buying results you already have. A per-item retry costs roughly <code>1 + error_rate \u00d7 attempts<\/code> of the base run; whole-job resubmission costs a full multiple per round.<\/li>\n<li><strong>Do not retry deterministic errors.<\/strong> A 400 or 422 means the request body or the schema is wrong. The same input will produce the same error every time. Fix the prompt, then submit those items as a <em>new<\/em> job with a new prompt hash \u2014 and route them through your dead-letter table so they are visible.<\/li>\n<li><strong>Cap attempts and record terminal failures.<\/strong> Three attempts is enough for transient errors. After that, write the item to a failed table with its last error so downstream consumers can decide between a fallback model and a human review.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Retries are billable, so they belong in your cost accounting rather than a log line nobody reads. If you are attributing spend per tenant, per pipeline, or per customer, the retry rounds have to land in the same ledger as the base run \u2014 which is exactly what <a href=\"https:\/\/qoraapi.com\/blog\/ai-usage-metering-billing\/\">metering AI usage<\/a> is for. Without it, a pipeline with a flaky 5% error rate looks 5% more expensive than it is, and nobody notices until the invoice arrives.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Throughput planning<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Batch removes concurrency management from your side but not from the provider's. Your ceiling is now a small set of hard limits: max requests per input file, max concurrent jobs per account, max items in flight, and the requests-per-minute cap on the realtime submit and poll calls themselves.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The planning math is one equation. If you must process <em>N<\/em> items per day, each job carries <em>C<\/em> items, and turnaround is <em>T<\/em> hours, the number of jobs you need in flight simultaneously is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>concurrent_jobs = (N * T \/ 24) \/ C\n\n# 10M items\/day, 50k items per job, 6h turnaround:\n# (10_000_000 * 6 \/ 24) \/ 50_000 = 50 concurrent jobs\n#\n# If the account cap is 20 concurrent jobs, your ceiling with this chunk size is:\n# 20 * 50_000 * 24 \/ 6 = 4M items\/day  ->  you are 2.5x short of the target.\n#\n# Fixes, in order of least pain:\n#   1. Shrink T (request a faster window, if one is offered)\n#   2. Raise the cap with the provider\n#   3. Shard across two accounts\/providers via a gateway\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Note what that equation implies: <strong>chunk size is a throughput lever, not just a failure-domain lever<\/strong>. Larger chunks mean fewer concurrent jobs for the same volume, but a single failure costs more items. The usual sweet spot is the largest chunk your retry budget can tolerate \u2014 if re-running one chunk is acceptable at your error rate, the chunk is not too big.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Do not forget the control plane. Submitting 50 jobs and polling each every 10 seconds is 5 requests per second of pure status traffic, and those calls hit the same per-key rate limits as your production traffic. Poll with exponential backoff (start at 30s, cap at 5 minutes for long windows), and expect the occasional 429 on a status check \u2014 the handling is the same as any other throttled call, covered in our guide to <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-rate-limits-429-errors\/\">rate limits<\/a> and 429 errors. A job that is throttled on <em>polling<\/em> has not failed; it is just being checked too eagerly.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Lever<\/th><th>Effect on throughput<\/th><th>Cost of pulling it<\/th><\/tr><\/thead><tbody><tr><td>Larger chunk size<\/td><td>Fewer concurrent jobs needed<\/td><td>Bigger blast radius per failure<\/td><\/tr><tr><td>Shorter completion window<\/td><td>Lower <em>T<\/em>, fewer jobs in flight<\/td><td>Smaller discount<\/td><\/tr><tr><td>More concurrent jobs<\/td><td>Linear gain, up to the account cap<\/td><td>Requires a provider-side raise<\/td><\/tr><tr><td>Dedupe before submit<\/td><td>Cuts <em>N<\/em> directly<\/td><td>None \u2014 pure win<\/td><\/tr><tr><td>Multi-provider sharding<\/td><td>Multiplies the cap<\/td><td>Two integrations, unless you use a gateway<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Running batch through a gateway<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Every provider implements batch slightly differently \u2014 different upload endpoints, different job status enums, different output schemas, different window names. Supporting three of them means three sets of submit\/poll\/collect code and three places for a bug to hide.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An OpenAI-compatible gateway collapses that. The same <code>build_jsonl \/ submit \/ poll \/ collect<\/code> functions from earlier run unchanged; you move between models by editing the <code>model<\/code> string inside each line's <code>body<\/code>. That makes two patterns practical that are painful otherwise:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Per-item model routing inside one run.<\/strong> Cheap items go to a small\/fast model, ambiguous items to a mid model \u2014 same file, same job, one polling loop. Split into per-model chunks only when you need per-model SLAs.<\/li>\n<li><strong>Uniform usage records.<\/strong> One invoice and one usage record per <code>custom_id<\/code> means per-tenant chargeback and retry accounting come from a single source instead of three dashboards.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">That is the specific problem an AI API relay solves for batch workloads: one key, many models, and a single job-submission shape in front of all of them. <a href=\"https:\/\/qoraapi.com\/\" target=\"_blank\" rel=\"noopener\">qoraapi.com<\/a> exposes many models behind one OpenAI-compatible endpoint, so the pipeline above does not need a provider-specific branch when you add a model or absorb an outage.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Is batch cheaper than realtime for the same model?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Yes \u2014 and it is the same model, so there is no quality penalty. Batch is a pricing tier for asynchronous delivery, typically landing around 0.4\u00d7\u20130.6\u00d7 the realtime rate. The measured saving is often larger, because batch also eliminates the retry waste a realtime fan-out generates when it hits rate limits.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How long does a batch job take?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You choose a completion window (commonly 24 hours, sometimes shorter) and the provider commits to finishing inside it. Actual p50 is usually far faster \u2014 tens of minutes to a few hours for typical job sizes \u2014 but you should schedule against the window, not the median. Treat the window as the worst case your pipeline is designed to absorb.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Can I use batch for streaming or interactive features?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. Batch returns results only after the job completes, so there is no partial token stream to forward and no way to answer a request that depends on the previous one. Interactive UX and agent loops need realtime calls; batch is for work where nothing is waiting.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What happens when a batch job expires?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Results for items that completed are still written to the output file, and the unfinished items appear as failures in the error file. Reconcile by <code>custom_id<\/code>, then resubmit only the missing ids \u2014 never the whole job. If expiry happens repeatedly, your chunk size is too large for the window you requested.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Batch is the highest-leverage cost lever available to an offline AI pipeline, and it is not a drop-in switch \u2014 it is an architecture. Decide with the three tests, then build so the pipeline can be re-run safely: deterministic chunks, content-derived job ids, state in a table, dedupe before you submit. Retry per item, never per job, and classify errors before you spend money on them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, size the system with the throughput equation before you launch, not after the first job misses its deadline. If you want the batch lifecycle in front of many models without writing a provider branch for each one, start from the <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-gateway-guide\/\">AI API gateway guide<\/a> and the <a href=\"https:\/\/qoraapi.com\/blog\/openai-compatible-api-guide\/\">OpenAI-compatible API explainer<\/a>, then point the code above at a single endpoint.<\/p>\n\n\n\n\n<h3 class=\"wp-block-heading\">Related reading<\/h3>\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/qoraapi.com\/blog\/reduce-ai-api-costs\/\">How to Reduce AI API Costs: A Practical Guide for Developers<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-api-rate-limits-429-errors\/\">How to Handle AI API Rate Limits and 429 Errors<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-usage-metering-billing\/\">Metering and Billing AI Usage Per User: A Practical SaaS Guide<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/document-data-extraction\/\">Extracting Structured Data from Documents with AI APIs<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/local-llm-vs-api\/\">Local LLMs vs API: A Real Cost and Latency Comparison for 2026<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/model-context-protocol-mcp\/\">What Is the Model Context Protocol (MCP)? Connect Your AI to Real Tools<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/connect-cursor-cline-continue-custom-api-endpoint\/\">How to Connect Cursor, Cline and Continue to a Custom AI API Endpoint<\/a><\/li><\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>Batch AI APIs trade latency for cost. Learn how to submit jobs, design idempotent chunked pipelines, handle partial failures, and plan throughput.<\/p>\n","protected":false},"author":1,"featured_media":135,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,6,9,7],"class_list":["post-136","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-api","tag-ai-api","tag-api-gateway","tag-developer-tools","tag-developers"],"_links":{"self":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/136","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/comments?post=136"}],"version-history":[{"count":2,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/136\/revisions"}],"predecessor-version":[{"id":262,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/136\/revisions\/262"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media\/135"}],"wp:attachment":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media?parent=136"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/categories?post=136"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/tags?post=136"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}