GPT-5.6 is here, and the bill is the first thing teams notice. OpenAI lists GPT-5.6 Sol at $4 input / $20 output per 1M tokens and GPT-5.6 Terra at $2 / $12. Run an agent or a coding assistant all day and those numbers add up fast. If you are searching for the cheapest GPT-5.6 API, the short answer is a pay-as-you-go gateway that resells the same models at a discount. This guide shows the exact math on Qora API, where GPT-5.6 runs at about 30% of the official price.
Cheapest GPT-5.6 API: official vs Qora prices
Official prices come from OpenAI’s model pages for gpt-5.6-sol, gpt-5.6-terra and gpt-5.5. Qora prices come from the live Qora pricing page: the default group multiplies the official rate by 0.3. All prices are per 1M tokens, text, prompts up to 272K tokens.
Note that OpenAI describes the Sol price as promotional until November 30, 2026. The Qora price is a fixed fraction of the official rate, so it follows whatever the official list price is. Check the models page for the current number before you budget.
What a real month looks like
Say your app sends 10M input tokens and 2M output tokens a month.
Model
Official monthly
Qora monthly
You keep
gpt-5.6-sol
$80.00
$24.00
$56.00
gpt-5.6-terra
$44.00
$13.20
$30.80
gpt-5.5
$110.00
$33.00
$77.00
Scale that to 100M input and 20M output and Sol drops from $800 to $240. The savings grow linearly with usage, which is why heavy agent workloads benefit most.
How to access GPT-5.6 through Qora in 5 minutes
Qora exposes an OpenAI-compatible endpoint, so any SDK or tool that lets you change the base URL works unchanged.
For editors and agents, the setup is identical: paste the base URL and key into the custom OpenAI endpoint field. Our walkthrough on using GPT-5 in Cursor with a custom endpoint applies to GPT-5.6 as well, just swap the model name. If you are comparing gateways first, read OpenRouter vs Qora API.
Sol or Terra? Pick the cheapest model that does the job
OpenAI positions Sol as the flagship for complex professional work and Terra as the balance of intelligence and cost. Terra costs half as much as Sol on both input and output, on Qora and officially.
Use Terra for classification, extraction, summaries, chat support, routine code edits and most agent sub-steps.
Use Sol for hard reasoning, long multi-file refactors and final-pass review.
Mix them. Route easy calls to Terra and escalate only when needed. This alone often cuts spend by 40% or more before any discount.
Five ways to cut your GPT-5.6 bill further
Control reasoning effort. GPT-5.6 supports effort levels from none to max, and OpenAI lists medium as the default. Higher effort means more reasoning tokens, billed as output. Drop to low for simple tasks.
Cap output length. Output is five to six times the price of input. Set max_tokens and ask for concise answers.
Stay under 272K tokens. OpenAI prices longer prompts at 2x input and 1.5x output for the whole request. Trim context or summarize old turns. See our guide to reducing AI API costs.
Keep prompts stable. A fixed system prompt prefix makes caching more effective. Our prompt caching guide explains how.
Set budgets. Track spend per key and stop runaway loops early.
FAQ
What is the cheapest way to use the GPT-5.6 API?
Based on the prices above, Qora’s default group charges about 30% of the official rate: $1.20 / $6.00 for Sol and $0.60 / $3.60 for Terra per 1M tokens.
Is the endpoint OpenAI-compatible?
Yes. Use https://qoraapi.com/v1 as the base URL with the standard OpenAI SDKs and tools.
Is there a subscription?
No. Qora is pay-as-you-go. You top up and pay for tokens you use.
Can I use other models with the same key?
Yes. Qora lists 350+ models, including Claude and Gemini, behind the same account. Browse them on the models page.
How do I get free credit to test?
New users can DM @QoraAPI on X or email [email protected] to claim a $5 trial credit.
Start with $5 free
Run your own prompts against GPT-5.6 Sol and Terra and compare the invoice. Create your Qora account, then DM @QoraAPI on X or email [email protected] for your $5 trial credit. Full rates are on the pricing page.
Aider is one of the best AI pair programmers you can run in a terminal. It maps your repo, edits files, and commits every change to git. It also burns through tokens. Every request carries a repo map, the files you added to the chat, and the conversation so far. Pair that with Claude Opus or Sonnet at official Anthropic rates and a busy week of refactoring can turn into a three-figure bill.
The good news: Aider talks to any OpenAI-compatible endpoint. That means you can keep the exact same Claude models and point Aider at a cheaper gateway. This guide shows how to run Aider with a cheap Claude API through Qora API, what it actually costs per model, and how to cut your token use even further.
Why Aider gets expensive with Claude
Aider’s token use comes from three places:
The repo map. Aider sends a compressed map of your codebase so the model knows where things live. On a large repo this alone is thousands of tokens per request.
Files in the chat. Every file you /add is sent in full on every turn until you /drop it.
Chat history. Long sessions keep re-sending earlier messages, so input tokens grow with every turn.
Input tokens usually outnumber output tokens many times over in a coding session. So the per-token input price matters most, and that is exactly where a discount pays off.
Claude API cost for Aider: official vs Qora
These are per-million-token prices. Official prices come from Anthropic’s pricing page; Qora prices come from the Qora pricing page. Every model below is cheaper on Qora than buying direct.
The newest Claude 5.5 models run on Qora’s Stable group at about 60% of the official price. The rest run on the default group at about 30% of official.
What that means for a month of Aider
Take an example month of heavy Aider use: 60M input tokens and 6M output tokens. Your own numbers will differ, but the ratio is typical of repo-map-heavy sessions.
Model
Official cost
Qora cost
Claude Opus 4.7
$450
$135
Claude Sonnet 4.6
$270
$81
Claude Sonnet 5
$180
$54
Claude Opus 5.5
$360
$216
Claude Haiku 4.5
$90
$27
Same model, same Aider workflow, a fraction of the bill. For the math behind input vs output pricing, see our guide to AI API pricing explained.
How to set up Aider with Qora API
Qora exposes an OpenAI-compatible endpoint at https://qoraapi.com/v1, and Aider supports OpenAI-compatible APIs out of the box. Setup takes about two minutes.
New user? Message @QoraAPI on X or email [email protected] to get $5 of free trial credit, enough to test Aider on a real repo.
Step 2: Install Aider
python -m pip install aider-install
aider-install
Step 3: Point Aider at Qora
Set your endpoint and key as environment variables:
# Mac / Linux
export OPENAI_API_BASE=https://qoraapi.com/v1
export OPENAI_API_KEY=your-qora-key
# Windows (restart the terminal afterwards)
setx OPENAI_API_BASE https://qoraapi.com/v1
setx OPENAI_API_KEY your-qora-key
Step 4: Start Aider with a Claude model
Prefix the Qora model ID with openai/ so Aider routes it through the OpenAI-compatible endpoint:
cd /path/to/your/project
aider --model openai/claude-sonnet-4-6
Aider may print a warning that it does not recognize the model’s context window or pricing. That is expected for custom endpoints and does not affect the requests. Add --no-show-model-warnings to hide it.
Step 5: Save it in a config file
To avoid typing flags every time, create .aider.conf.yml in your project or home directory:
Do not commit this file if it holds your key. Add it to .gitignore, or keep the key in an environment variable instead.
Cost-cutting tips for Aider
1. Use a cheap weak model
Aider uses a “weak model” for side jobs like writing commit messages and summarizing chat history. Setting it to openai/claude-haiku-4-5-20251001 ($0.30 / $1.50 on Qora) keeps those calls nearly free.
2. Use architect mode with two models
Architect mode lets one model plan and another apply the edits. Plan with a strong model and edit with a cheaper one:
Opus handles the hard reasoning; Sonnet writes the diffs at $0.90 / $4.50 instead of Opus rates.
3. Keep the chat lean
/add only the files the change needs, and /drop them when you are done.
Run /clear between unrelated tasks so old history stops riding along.
Use /tokens to see what is eating your context before it eats your budget.
4. Match the model to the task
Most day-to-day edits, tests and small refactors run well on Sonnet 4.6 or Sonnet 5. Save Opus for gnarly multi-file changes. For a broader playbook, read how to reduce AI API costs and our guide to choosing the right model.
5. Top up as a partner if you spend a lot
Qora is pay-as-you-go with no subscription. Partners get an extra 30% on every top-up: put in $1,000 and get $1,300 of credit.
Does Aider work with Claude through an OpenAI-compatible API?
Yes. Aider supports any OpenAI-compatible endpoint. Set OPENAI_API_BASE to https://qoraapi.com/v1 and use model names like openai/claude-sonnet-4-6.
What is the cheapest Claude model for Aider?
Of the models in the table above, Claude Haiku 4.5 is the cheapest at $0.30 / $1.50 per million tokens on Qora. For serious coding, Claude Sonnet 5 at $0.60 / $3.00 is the best value among the stronger models.
Is it the same Claude model?
You call the same model IDs, such as claude-opus-4-7 and claude-sonnet-4-6. Qora is a gateway, so you get one key for Claude, GPT, Gemini and 350+ other models.
Can I use Qora with tools that need the Anthropic API format?
Yes. Claude models also have an Anthropic-compatible endpoint with base URL https://qoraapi.com. Aider itself only needs the OpenAI-compatible one.
Do I need a subscription?
No. Qora is pay-as-you-go. You only pay for the tokens you use. See full rates on the pricing page or browse every model at qoraapi.com/models.
Start using Aider for less
Aider is worth every token. You just do not need to pay full price for them. Create your Qora account, set two environment variables, and keep coding with Claude at up to 70% off official rates. Want to try it first? Message @QoraAPI on X or email [email protected] and ask for your $5 free trial credit.
Claude Code is one of the best coding agents available, but on pay-as-you-go API pricing it can burn through $20 to $100 a day on a busy codebase. This guide shows how to point Claude Code at a cheaper Anthropic-compatible endpoint so you keep the same models and workflow for about 70% less.
Why Claude Code costs so much
An agent loop rereads files, runs tools and resends its growing context on every step. A single feature can take 50 to 150 model calls, each sending tens of thousands of input tokens. At official Claude Opus prices ($5 per million input tokens, $25 per million output), a day that uses 8 million input and 400,000 output tokens costs about $50.
The fix: change the base URL
Claude Code reads two environment variables that let you send its requests to any Anthropic-compatible endpoint: ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN. Qora API exposes Claude models on an Anthropic-compatible endpoint at around 30% of official prices, so the same $50 day costs about $15.
Create a key. Sign up at qoraapi.com and create an API key in the console. New users can get $5 of free credit by DMing @QoraAPI on X or emailing [email protected].
On Windows PowerShell use $env:ANTHROPIC_BASE_URL="https://qoraapi.com" and so on.
Run Claude Code as usual with claude. Add the exports to your ~/.zshrc or ~/.bashrc to make them permanent.
Prefer a config file? Put the same values under "env" in ~/.claude/settings.json.
Get even more out of every dollar
Use Sonnet as your default and switch to Opus with /model only for hard architecture or debugging work.
Compact early. Run /compact when a session gets long so the agent stops resending stale context.
Keep CLAUDE.md short. It is sent with every request, so every extra paragraph costs money all day.
Clear between tasks./clear starts a fresh context instead of dragging the last feature along.
Works with other coding agents too
The same key works through the OpenAI-compatible endpoint https://qoraapi.com/v1 in Cursor, Cline, Roo Code, Continue, Aider and any OpenAI SDK, so one balance covers every tool you use.
FAQ
Is it the same Claude model?
Yes. Requests go to the Claude model you name, through Qora API’s gateway.
Do I need a subscription?
No. It is pay as you go, with no monthly fee. Unused balance is refundable on request.
What if a request fails?
Open a ticket in the console or email [email protected]. Support runs 24/7 with an average first reply within 30 minutes.
Claude is the favorite model for roleplay in SillyTavern, and it is also the one that drains your wallet fastest. Long character cards, lorebooks and chat history mean every reply resends thousands of tokens. This guide shows how to keep using Claude (and GPT or Gemini) in SillyTavern for about 70% less by connecting it to an OpenAI-compatible gateway.
Why SillyTavern gets expensive
SillyTavern sends the whole prompt on every message: system prompt, character card, persona, lorebook entries, example dialogue and as much chat history as fits in your context setting. A typical roleplay turn with a 16k context easily sends 10,000 to 15,000 input tokens and gets back 300 to 800 output tokens.
At official Claude Opus prices ($5 per million input tokens, $25 per million output), a session of 200 messages at 12,000 input and 500 output tokens costs about $14.50. Heavy users run several of those a week.
The cheaper route: an OpenAI-compatible gateway
Qora API gives you one API key for 350+ models, with Claude Opus, Sonnet and Haiku, GPT and Gemini at around 30% of official prices. Because it speaks the standard OpenAI format, SillyTavern connects to it with the built-in Custom (OpenAI-compatible) source. No extensions needed.
Get a key. Create an account at qoraapi.com, open the console and create an API key. New users can get $5 of free credit by DMing @QoraAPI on X or emailing [email protected].
Open API Connections. In SillyTavern, click the plug icon at the top.
Choose the API type. Set API to Chat Completion, and Chat Completion Source to Custom (OpenAI-compatible).
Enter the endpoint. In Custom Endpoint (Base URL) enter https://qoraapi.com/v1.
Paste your key into Custom API Key.
Pick a model. Click Connect, then choose a model from the list or type its ID, for example claude-opus-4-7, claude-sonnet-4-6 or deepseek-chat.
Send a test message. If you get a reply, you are done.
Five more ways to cut your SillyTavern bill
Trim your context. Dropping context from 32k to 16k tokens roughly halves the input cost of every message, and most chats never need more.
Use Summarize. The built-in Summarize extension replaces old history with a short summary, so long chats stay cheap.
Keep lorebooks lean. Set entries to trigger on keywords instead of always-on.
Mix models. Use Haiku or DeepSeek for casual scenes and switch to Opus or Sonnet for the moments that matter.
Cap response length. Output tokens cost about five times more than input tokens on Claude, so a sensible max response length saves real money.
FAQ
Is it the real Claude?
Yes. You choose the model by name and your request is routed to that model.
Do I need a subscription?
No. Top up any amount and pay only for the tokens you use. Unused balance is refundable on request.
Does it work with other frontends?
Yes. Anything that accepts an OpenAI-compatible endpoint works the same way, including Agnai, RisuAI and LibreChat.
Looking for a cheap Gemini API? The cheapest way to call Gemini on Qora is gemini-2.5-flash at $0.12 input and $0.48 output per 1 million tokens. The stronger model, gemini-2.5-pro-preview-06-05, costs $0.75 input and $6.00 output per 1 million tokens. Both work through one OpenAI-compatible API key at https://qoraapi.com/v1 — no Google Cloud account, no separate Gemini billing, no code rewrite.
This guide shows you the exact model names to use, the exact prices, working code you can copy, and how to pick Pro or Flash for each job so you stop paying Pro prices for Flash work. All prices in this article are Qora Model Square prices, checked on October 27, 2026.
How Much Does the Gemini API Cost on Qora?
Gemini pricing is per token. You pay separately for input tokens (the text you send) and output tokens (the text the model writes). Output costs more than input on every Gemini model, so long answers are the expensive part of any Gemini bill.
Model
Input (per 1M tokens)
Output (per 1M tokens)
gemini-2.5-flash
$0.12
$0.48
gemini-2.5-flash-preview-04-17
$0.18
$0.72
gemini-2.0-flash
$0.18
$0.72
gemini-2.5-pro-preview-03-25
$0.375
$3.00
gemini-2.5-pro-preview-06-05
$0.75
$6.00
gemini-2.5-pro-preview-05-06
$0.75
$6.00
Two things matter here. First, there is no plain gemini-2.5-pro model ID on the Model Square. The Pro model is listed under dated preview names, and the one used in this article is gemini-2.5-pro-preview-06-05. If you send gemini-2.5-pro as the model name, the request will fail with a model-not-found error. Second, the two Pro previews do not cost the same: the 03-25 snapshot is $0.375 input and $3.00 output, exactly half the price of the 06-05 and 05-06 snapshots.
A worked example makes the gap concrete. Suppose you send 1 million input tokens and generate 200,000 output tokens. On gemini-2.5-flash, that costs $0.12 + $0.096 = $0.216, about 22 cents. On gemini-2.5-pro-preview-06-05, the same work costs $0.75 + $1.20 = $1.95. Same tokens, nine times the price.
Gemini 2.5 Pro vs Flash: Which One for Which Job?
Flash and Pro are not two versions of the same tool. They are priced for different jobs, and using Pro for a Flash job is the most common way people overpay for Gemini.
Use Flash for high-volume, routine work: classifying tickets, extracting fields from invoices, summarising short documents, and drafting first-pass replies. Flash costs $0.12 per million input tokens, so 10 million input tokens costs $1.20.
Use Pro when the task needs deeper reasoning or a long, tangled input: debugging a multi-file problem, analysing a 50-page contract, or planning a multi-step task. Pro costs $0.75 per million input tokens on the 06-05 snapshot, and its output costs $6.00 per million tokens.
A practical rule: start every feature on Flash. Run it against your real inputs and check the outputs. If Flash gets the answer right, you are done, and your Gemini bill stays small. If Flash fails on a specific kind of input — long documents, multi-step reasoning, tricky edge cases — route only that kind of input to Pro. For the general routing playbook, see How to Reduce AI API Costs: A Practical Guide for Developers.
Set Up Gemini with One API Key
You need a Qora API account and key. Sign up at https://qoraapi.com/, create a key in the Console, and top up your balance. Qora is pay-as-you-go in USD, with a minimum top-up of $50. If your balance is zero, your first request will fail even if the key is correct, so check the balance in the Console first.
Test with cURL
This calls gemini-2.5-flash through the Qora endpoint. Replace $QORA_API_KEY with your key, or export it first with export QORA_API_KEY="sk-...".
curl https://qoraapi.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $QORA_API_KEY" \
-d '{
"model": "gemini-2.5-flash",
"messages": [
{"role": "user", "content": "Summarise this in three bullet points: Gemini is fast and cheap."}
],
"max_tokens": 300
}'
A successful response returns a JSON object with choices[0].message.content. If you get 401, the key is wrong or missing. If you get 404, the model name is wrong — copy it exactly from the Model Square. If you get 429, you hit a rate limit; wait and retry.
Use Gemini from Python
The same call works with the OpenAI Python SDK, because Qora is OpenAI-compatible. Only the base URL and the model name change.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://qoraapi.com/v1",
api_key=os.environ["QORA_API_KEY"],
)
response = client.chat.completions.create(
model="gemini-2.5-flash",
messages=[{"role": "user", "content": "List three uses for Gemini Flash in one sentence each."}],
max_tokens=300,
)
print(response.choices[0].message.content)
For a chatbot or a live interface, stream the response instead of waiting for the full answer. Streaming does not change the price — you pay for the same tokens — but the user sees the first words in under a second instead of waiting for the whole reply.
stream = client.chat.completions.create(
model="gemini-2.5-flash",
messages=[{"role": "user", "content": "Give me three tips for saving on API costs."}],
max_tokens=300,
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
How to Cut Your Gemini Bill Further
Routing to Flash is the biggest saving. Three smaller habits add up after that.
First, ask for less output. Output tokens cost four times as much as input tokens on Flash ($0.48 vs $0.12) and eight times as much on Pro ($6.00 vs $0.75). A vague prompt such as “tell me about this document” produces a long answer and a large bill. A specific prompt produces a short answer: “Return a JSON object with three fields — summary, key_dates, and action_items. No other text.” Set max_tokens as a hard cap, and treat a response that hits the cap as a signal to split the task.
Second, keep repeated context stable. If every request starts with the same system prompt, instructions, or reference document, do not rewrite it on every call and do not shuffle its order. Stable, repeated input is what makes prompt caching work when the provider offers it. This article does not quote a Gemini cache discount, because no Gemini cached-input price was verified for this guide — check the Model Square for the current cached-input price on the model you use before relying on a specific number.
Third, batch work that can wait. Summarising documents, labelling data, and generating nightly reports do not need an answer in seconds. If your provider offers a lower price for non-urgent batch requests, use it for that work and keep Flash for live traffic. For more on batching, see the 2026 AI API Cost Index.
Gemini vs Claude, GPT, and DeepSeek Prices
Gemini Flash is cheap, but it is not the cheapest model on Qora for every job. Here is how it compares with the alternatives named in this article, per 1 million tokens:
Model
Input
Output
Cheapest for
gemini-2.5-flash
$0.12
$0.48
High-volume routine text work
claude-3-5-haiku-20241022
$0.15
$0.75
Short, fast tasks where Claude quality matters
deepseek-chat
$0.30
$1.20
Cheap general chat and code
gpt-5
$0.45
$3.60
General work where GPT behaviour is expected
gemini-2.5-pro-preview-06-05
$0.75
$6.00
Hard reasoning and long documents
claude-3-5-sonnet-20241022
$0.90
$4.50
Complex code and long-form writing
The practical takeaway: Flash beats Haiku on price for routine work ($0.12 vs $0.15 input), and both are far cheaper than Sonnet or GPT-5. But if your task needs Claude’s writing style or GPT’s tool behaviour, the cheapest model that fails the task is the most expensive one — you pay twice, once for the failure and once for the retry. Test Flash on your real inputs first. For a direct gateway comparison, see OpenRouter vs Qora API: Which Cheap AI API Gateway Is Better in 2026?
Three Mistakes That Waste Money on Gemini
The first mistake is the model name. gemini-2.5-pro does not exist on the Model Square, and neither does gemini-pro or gemini-flash on its own. The working names are gemini-2.5-flash and gemini-2.5-pro-preview-06-05 (or another dated preview). A 404 error almost always means the name is wrong.
The second mistake is the base URL. It must be exactly https://qoraapi.com/v1, with no trailing slash and no /chat/completions path. Clients add the rest of the path themselves. If you paste https://qoraapi.com/v1/chat/completions as the base URL, the client builds a doubled path and every request fails.
The third mistake is running Pro as the default. It is tempting to set Pro everywhere “to be safe”. On the 06-05 snapshot, 1 million input tokens cost $0.75 on Pro against $0.12 on Flash, and output costs $6.00 against $0.48. A feature that sends 10 million input tokens a day costs $7.50 a day on Pro and $1.20 a day on Flash. Default to Flash, escalate to Pro only where Flash fails.
FAQ
What is the cheapest Gemini API?
On Qora, gemini-2.5-flash is the cheapest Gemini model in this guide: $0.12 input and $0.48 output per 1 million tokens, checked on the Model Square on October 27, 2026. For harder tasks, gemini-2.5-pro-preview-03-25 costs $0.375 input and $3.00 output — half the price of the newer Pro preview.
How much does Gemini 2.5 Pro cost per 1M tokens?
On Qora, gemini-2.5-pro-preview-06-05 costs $0.75 input and $6.00 output per 1 million tokens. The older snapshot gemini-2.5-pro-preview-03-25 costs $0.375 input and $3.00 output. There is no plain gemini-2.5-pro model ID on the Model Square — use the exact dated preview name.
Is Gemini 2.5 Flash good for coding?
Flash is good for routine coding tasks: small fixes, explanations, formatting, and simple generation. For complex refactoring or multi-file debugging, Pro is the better choice. Test Flash on your real tasks first and route only the hard cases to Pro.
Can I use Gemini with the OpenAI SDK?
Yes. Qora is OpenAI-compatible, so the OpenAI Python and Node.js SDKs work with Gemini models. Set base_url to https://qoraapi.com/v1, use your Qora API key, and set the model name to gemini-2.5-flash or a Pro preview ID.
What is the Gemini model name on Qora?
Copy it exactly from the Model Square. The working names in this guide are gemini-2.5-flash and gemini-2.5-pro-preview-06-05.
Try It
Get $5 free trial credit on request — contact us at [email protected] and we’ll add $5 to your account. Sign up at https://qoraapi.com/, create one API key, and try the same prompt on gemini-2.5-flash and gemini-2.5-pro-preview-06-05 to see the price and quality difference for yourself. The full client setup is in the docs at https://qoraapi.com/docs/.
Cursor is one of the fastest ways to write code with AI. It usually works with its own built-in models. But you can also point Cursor at a custom API endpoint, and use a model of your choice, including GPT-5, through a single API key.
This guide shows you how to do that with Qora API, step by step, with a test to confirm it works, and fixes for the two errors you are most likely to hit: 401 and 429.
A note before you start. The exact GPT-5 model name to use is the one shown on the Qora Model Square. Model names change as new versions are released, and this article does not invent a model name. If the name in this guide does not work, check the Model Square list first and use the GPT-5 name shown there. GPT-5 on Qora costs $0.45 per 1M input tokens and $3.60 per 1M output tokens, as shown on the Model Square (checked 2026-10-27). Prices can change, so check the Model Square for the current price before you use GPT-5 heavily.
What You Need Before You Start
You need three things:
Cursor, installed and updated.
A Qora API account and API key. Sign up at https://qoraapi.com/, create a key in the Console, and copy it. Keep the key secret.
Enough balance on your Qora account to run a few test requests. Get $5 free trial credit on request — contact us at [email protected] and we’ll add $5 to your account. Trial credit is not automatic, so email first if you want to try before topping up.
Step-by-Step Setup
Step 1: Get Your Qora API Key
Open https://qoraapi.com/ in your browser and sign in. Go to the Console and find the API Keys section. Create a new key, give it a name you will recognise, such as cursor-gpt5, and copy the key value.
Copy the key once and store it somewhere safe, such as a password manager. Many key screens only show the full key once. If you lose it, you can create a new key and delete the old one, but you cannot recover the old key value.
Do not use a key from another service here. The key must be a Qora API key, because Cursor will send it to Qora’s endpoint. A key from OpenAI, Anthropic, or another gateway will not work with Qora’s endpoint, and will return a 401 error.
Step 2: Open Cursor Settings
Open Cursor. Go to Settings. You can usually get there from the gear icon, or from the Cursor menu, depending on your operating system.
In Settings, find the Models section. This is where Cursor lists the models it can use, and where you add a custom model.
Step 3: Add GPT-5 as a Custom Model
In the Models section, look for an option to add a custom model, or to use a custom API endpoint. The label is often something like “Add Model”, “Custom Model”, or “Override OpenAI Base URL”. If you cannot find it, search the Settings page for “API key” or “base URL”.
Enter these three values:
Base URL:https://qoraapi.com/v1
API key: your Qora API key, pasted from Step 1
Model name: the GPT-5 model name exactly as shown on the Qora Model Square
The base URL must be exactly https://qoraapi.com/v1, with no trailing path such as /chat/completions, and no trailing slash. Cursor adds the rest of the path itself. If you add /chat/completions to the base URL, Cursor will build a wrong URL and the request will fail.
The model name matters more than it looks. Cursor sends the model name to Qora, and Qora uses it to pick the model. If the name is wrong, Qora will reject the request. Use the GPT-5 name exactly as it appears on the Model Square, including any version suffix. Do not guess a name, and do not use a name from a different provider’s list.
Save the settings. Cursor may test the connection when you save. If it shows a success message, go to the next step. If it shows an error, do not move on. Go to the troubleshooting section at the end of this guide first.
Open a new chat in Cursor. Select the GPT-5 model you just added from the model picker. If you cannot see it in the picker, go back to Settings and check that the custom model is enabled, not just added. Some Cursor versions require you to enable a model before it appears in the picker.
Type a short test prompt, such as:
Write a Python function that reverses a string, and add a one-line comment explaining it.
Send the message and wait for the reply. A short reply that arrives in a few seconds means the setup works. The model, the key, and the base URL are all correct.
Now try a coding task in your own project. Open a file, select a piece of code, and ask Cursor to explain it or to refactor it. This is the real test. A chat test proves the connection works. A code test proves the model is useful for your work.
Step 5: Use GPT-5 for Real Work, and Watch the Cost
Once the test works, you can use GPT-5 in Cursor the same way you would use any other model: chat, inline edits, and agent mode. Keep your requests focused. Select the code you want the model to change, and ask for one thing at a time. A vague request such as “improve this file” will generate a long answer and a large bill. A focused request such as “fix the off-by-one error in this loop” will generate a short answer and a small bill.
Check your Qora usage regularly. The Console shows your usage and your balance. If the balance drops faster than you expect, check which model you are using and how long your answers are. The cost of a coding assistant is mostly output tokens, and long answers are expensive on any model.
The setup in this guide is not specific to Cursor. Cline and Continue use the same OpenAI-compatible pattern: a base URL, an API key, and a model name.
To use GPT-5 in Cline or Continue, use the same three values: base URL https://qoraapi.com/v1, your Qora API key, and the GPT-5 model name exactly as shown on the Model Square. The menus are different, but the values are the same.
Troubleshooting: 401, 429, and Other Errors
401 Unauthorized
A 401 error means the API key was rejected. The most common causes are a key pasted with a space at the start or end, a key from a different service, a key that was deleted in the Qora Console, or an account with no balance. Check the key in Cursor settings, make sure it is a Qora API key, and check your balance in the Qora Console.
If the key is correct and the balance is positive, try the same key with a direct cURL call to check whether the problem is in Cursor or in the key:
In this example, gpt-5 is the GPT-5 model name as shown on the Model Square. If the name shown there changes, use the current name instead. If this call works and Cursor does not, the problem is in Cursor’s settings. If this call also returns 401, the problem is the key or the balance.
A 429 error means you have hit a rate limit. You sent too many requests too quickly, or too many tokens in a short time. Wait a few seconds and try again. If you are running an agent loop, add a short delay between requests, or reduce the number of requests by batching your work. Do not fix a 429 by creating a new key or a new account. The limit is on usage, not on the key.
If Cursor says the model was not found, the model name is wrong. Go to the Qora Model Square and copy the GPT-5 model name exactly as it appears there. Paste it into Cursor, save, and try again. Do not guess a model name. The Model Square is the source of truth.
Empty or Truncated Replies
If replies are empty or cut off, check the max_tokens setting. A very low limit will cut off the answer. Raise the limit a little and try again.
FAQ
Can I use GPT-5 in Cursor without paying OpenAI directly?
Yes. With Qora API, you use GPT-5 through Qora’s OpenAI-compatible endpoint at https://qoraapi.com/v1. You pay Qora, not OpenAI directly. You need a Qora API key and enough balance on your Qora account. GPT-5 costs $0.45 per 1M input tokens and $3.60 per 1M output tokens on Qora, as shown on the Model Square (checked 2026-10-27). Prices can change, so check the Model Square for the current price.
What base URL should I use for GPT-5 in Cursor?
Use https://qoraapi.com/v1 exactly, with no trailing path and no trailing slash. Cursor adds the rest of the path itself. If you add /chat/completions to the base URL, the request will fail.
What is the GPT-5 model name in Cursor?
Use the GPT-5 model name exactly as shown on the Qora Model Square. Model names change as new versions are released, and this article does not invent a model name. If the name in a guide does not work, check the Model Square and copy the name from there.
Why do I get a 401 error in Cursor?
A 401 error usually means the API key is wrong, is from a different service, was deleted, or has no balance. Check the key in Cursor settings, make sure it is a Qora API key, and check your balance in the Qora Console. Test the same key with a direct cURL call to see whether the problem is in Cursor or in the key.
Does the same setup work in Cline and Continue?
Yes. Cline and Continue use the same OpenAI-compatible pattern: base URL https://qoraapi.com/v1, your Qora API key, and the GPT-5 model name from the Model Square. The menus are different, but the values are the same.
Try It
Get $5 free trial credit on request — contact us at [email protected] and we’ll add $5 to your account. Sign up at https://qoraapi.com/, create one API key, and follow the steps above to use GPT-5 in Cursor. Start with a short test prompt, confirm it works, and then try it on your own code.
If you build with AI models, you have probably hit the same wall: every provider wants its own account, its own API key, and its own billing. OpenAI for GPT, Anthropic for Claude, Google for Gemini, xAI for Grok. Four accounts, four keys, four bills, and a code change every time you want to try a different model.
An AI API gateway solves that problem. One key, one endpoint, many models. OpenRouter is the best-known gateway of this kind. Qora API is a newer one with the same idea. This article compares the two fairly, so you can pick the one that fits your project.
A note on fairness before we start. This article does not quote OpenRouter prices, because none were verified for this guide. OpenRouter’s pricing, free models, and rate limits change often, and quoting a number from memory would be misleading. Where a claim about OpenRouter is not verified here, this article says so. Qora facts in this article come from Qora’s own Model Square and site, checked on 2026-10-27. Prices can change, so check both sites before you commit to either one.
What Is an AI API Gateway?
An AI API gateway is a single endpoint that sits in front of many model providers. You send a request to the gateway, the gateway forwards it to the provider you asked for, and it returns the answer. You pay the gateway, not each provider separately.
The benefit is not just billing. A gateway lets you switch models by changing one word in your code, the model name. It lets you fall back to a second model when the first one is down or rate-limited. And it lets you compare models on the same task without rewriting your integration for each provider.
OpenRouter is a well-known AI API gateway. It offers access to a large catalogue of models from many providers through one API key and an OpenAI-compatible endpoint. Developers use it to try new models quickly, to route requests between providers, and to avoid managing a separate account for every provider.
OpenRouter is a large, established service with a public model list and a community of users. That size is a real advantage: when a new model is released, it often appears on OpenRouter quickly, and there is usually documentation and community discussion about how to use it.
What this article does not claim about OpenRouter: its exact per-token prices, whether any particular model is free, what its rate limits are, or what its uptime is. None of those were verified for this guide. If price is your deciding factor, compare the two services on the same models, on the same day, using the prices listed on each site. Do not rely on a price quoted in a blog post, including this one.
Qora API at a Glance
Qora API is an OpenAI-compatible AI API gateway. One API key works for GPT, Claude, Gemini, Grok, and 350+ models through one endpoint: https://qoraapi.com/v1. Pricing is pay-as-you-go, in USD, per token. There is no subscription. The minimum top-up is $50.
Qora’s Model Square lists 354 models, with per-token input and output prices shown for each one. For example, Claude Sonnet 4.5 is listed at $0.90 input and $4.50 output per 1 million tokens, and claude-3-5-haiku-20241022 is listed at $0.15 input and $0.75 output per 1 million tokens. Those prices were checked on 2026-10-27. Always check the Model Square for the current price, because prices can change.
Qora also offers a $5 free trial credit on request. Get $5 free trial credit on request — contact us at [email protected] and we’ll add $5 to your account. Trial credit is not automatic, and it is not described as requiring no card. If you want to try Qora before topping up, email first and the credit will be added to your account.
Qora is a smaller and newer service than OpenRouter. That is a trade-off, and it is honest to say so. A smaller service may have fewer community resources, fewer third-party integrations, and less public track record. What it offers in return is a focused catalogue, a single pay-as-you-go price list, and direct ticket support, 24/7, with an average first reply within 30 minutes. This article makes no uptime or latency claim for either service, because none was verified. If a model is critical to your product, build a fallback to a second model, as explained in How to Build a Multi-Provider AI Failover Layer for 99.9% Uptime.
One Key and One Endpoint
Both services give you one key and one OpenAI-compatible endpoint. That is the core promise of a gateway, and both deliver it.
On Qora, the endpoint is https://qoraapi.com/v1, and the setup is the standard OpenAI client pattern:
OpenRouter uses the same pattern, with its own base URL and key. If your code already works with one OpenAI-compatible gateway, switching to the other is mostly a matter of changing the base URL, the key, and the model name. That portability is the point. You are not locked into either service by your code, and you can test both with the same application.
Qora lists 354 models on its Model Square, including GPT, Claude, Gemini, Grok, Qwen, DeepSeek, and others. The list is shown with prices, so you can compare models before you call them.
OpenRouter also lists a large catalogue of models from many providers. This article does not state an exact model count for OpenRouter, because that number changes as models are added and removed, and it was not verified for this guide. Both services cover the major providers, so model choice is unlikely to be the deciding factor for most projects. The question that matters more is whether the specific models you need are available on the service you pick, at a price you accept, on the day you need them.
Before you choose, make a short list of the models your application actually uses. Check each one on both services. A gateway that lists 350 models is no help if the one model you need is missing, and a gateway that lists 100 models is plenty if it has the three you use every day.
Pricing: Pay as You Go, in USD
Qora pricing is pay-as-you-go, in USD, per token. You top up your balance and spend it as you use models. There is no monthly subscription. The minimum top-up is $50, and top-up amounts of $50, $100, $500, $1000, $5000, and $10000 carry discounts of 0%, 1%, 3%, 7%, 12%, and 20% respectively. Payment is by Visa, Mastercard, Apple Pay, or Google Pay. Unused balance is refundable on request.
OpenRouter also charges per token, and its prices vary by model and provider. This article does not compare per-token prices model by model, because OpenRouter’s prices were not verified for this guide. That comparison is worth doing yourself, on the models you actually use, because a gateway that is cheap for GPT may not be cheap for Claude, and vice versa.
One honest difference to note is the minimum top-up. Qora’s $50 minimum is higher than some gateways that let you start with a few dollars. If you want to spend $5 to test an idea, Qora’s trial credit on request is the way to do it, not a $5 top-up. If you are a hobbyist who wants to spend a few dollars at a time, that minimum matters. If you are building a product and expect to spend $50 or more, it does not.
Which One Should You Choose?
There is no single right answer. Use the decision guide below.
Choose OpenRouter if you want the largest, most established gateway, the widest community, and the fastest access to brand-new models, and you are comfortable comparing prices yourself on the models you use.
Choose Qora API if you want a single pay-as-you-go price list in USD, a $5 trial credit on request, direct ticket support, and a focused catalogue of 350+ models through one OpenAI-compatible endpoint, and the $50 minimum top-up is not a problem for you.
Try both if you can. The switching cost is low, because both use the OpenAI-compatible pattern. Run the same requests on the same models, compare the price, the latency, and the quality, and pick the one that is cheaper and more reliable for your actual workload. That test is worth more than any comparison article, including this one.
Yes. Qora API is an OpenAI-compatible AI API gateway, like OpenRouter. One key gives you access to GPT, Claude, Gemini, Grok, and 350+ models through https://qoraapi.com/v1. If you are looking for an OpenRouter alternative, Qora is one option to test, alongside other gateways.
Is Qora API cheaper than OpenRouter?
It depends on the model. Qora lists its per-token prices on the Model Square, for example Claude Sonnet 4.5 at $0.90 input and $4.50 output per 1 million tokens. This article does not quote OpenRouter prices, because none were verified for this guide. Compare the two services on the exact models you use, on the same day, before deciding on price.
Does Qora API have a free trial?
Qora offers a $5 free trial credit on request. Get $5 free trial credit on request — contact us at [email protected] and we’ll add $5 to your account. The credit is not automatic. This article does not make a claim about OpenRouter’s free tier, because it was not verified for this guide.
What is the minimum top-up on Qora API?
The minimum top-up is $50, in USD. Top-up amounts of $50, $100, $500, $1000, $5000, and $10000 carry discounts of 0%, 1%, 3%, 7%, 12%, and 20% respectively. Unused balance is refundable on request.
Can I switch from OpenRouter to Qora API without rewriting my code?
In most cases, yes. Both services use the OpenAI-compatible pattern, so you mainly change the base URL, the API key, and the model name. Check the model names on the Qora Model Square first, because model names differ between gateways.
Try It
Get $5 free trial credit on request — contact us at [email protected] and we’ll add $5 to your account. Sign up at https://qoraapi.com/, create one API key, and run the same prompt on the same model through Qora and through your current gateway. Compare the price and the result, and keep the one that is better for your workload.
Claude is one of the best models for coding, writing, and reasoning. It is also one of the easiest models to overspend on. A long conversation, a large codebase, or a chatty agent loop can turn a small project into a surprisingly large API bill.
The good news is that you do not need a cheaper Claude. You need a cheaper way to call Claude. Most of the cost of a Claude API call comes from how you use it: which model you pick, how much context you send every time, and whether you pay full price for repeated input. Change those three things and the same work can cost a fraction of what it cost before.
This guide shows how to do that with one API key. All prices in this article are Qora Model Square prices, checked on 2026-10-27. Prices can change, so always check the Model Square before you build a budget around them.
How Much Does the Claude API Actually Cost?
Claude pricing is per token. You pay separately for input tokens, the text you send, and output tokens, the text the model generates. Output tokens cost several times more than input tokens on every Claude model, so a model that writes long answers is expensive even if its input price looks low.
Here are the Qora Model Square prices for the Claude models named in this article, per 1 million tokens:
Model
Input (per 1M tokens)
Output (per 1M tokens)
Claude Sonnet 4.5
$0.90
$4.50
claude-3-5-sonnet
$0.90
$4.50
claude-3-5-haiku-20241022
$0.15
$0.75
Those are the only Claude prices quoted in this article, because they are the only ones verified on the Model Square at the time of writing. Claude Opus is the largest and most capable Claude tier, and it costs more than Sonnet. This article does not quote an Opus price, because no Opus price was verified for this guide. Check the Model Square for the current Opus price before you use it.
Two things jump out from the table. First, Sonnet costs six times as much as Haiku on input and output. Second, output is five times the price of input on both models.
A quick example makes the gap concrete. Suppose you send 1 million input tokens and generate 200,000 output tokens. On claude-3-5-haiku-20241022, that costs about $0.30 in total. On Claude Sonnet 4.5, the same work costs about $1.80 in total. Same tokens, six times the price. The work did not change. The model choice did.
Pick the Smallest Model That Can Do the Job
The single biggest saving available to most developers is also the simplest: stop using Sonnet for everything.
Haiku is fast and cheap, and it is good enough for a large share of real work. Use Haiku for classification, extraction, formatting, summarisation, simple question answering, and routing decisions. Use Sonnet when the task actually needs deeper reasoning, long-form writing, or complex code generation. Use Opus only for the hardest tasks, where a weaker model fails repeatedly and the cost of failure is higher than the cost of the model.
A practical way to decide is to test with Haiku first. Build the feature with Haiku, run it against your real inputs, and check the output quality. If Haiku passes, you have just cut your Claude bill by roughly 80 percent. If Haiku fails on a specific kind of input, route only that kind of input to Sonnet. Most applications have a mix: 80 percent of requests are easy, 20 percent are hard. Paying Sonnet prices for the easy 80 percent is pure waste.
This is called model routing, and it is the difference between a cheap Claude API setup and an expensive one. You do not need a complex system to start. A simple rule in your code is enough: if the request is short and the task is routine, use Haiku; if it is long or complex, use Sonnet. You can refine the rule later. For a deeper look at routing in general, see How to Choose the Right AI Model: A Practical Model-Routing Guide.
Stop Paying for the Same Input Twice
The second biggest saving is caching. In most Claude applications, the same large input is sent again and again. A system prompt, a set of instructions, a project README, or a long document is included in every request, and you pay for it every time.
Prompt caching lets the provider reuse that repeated input instead of charging you full price for it again. If your system prompt and context do not change between requests, the cached part is much cheaper than fresh input. The exact discount depends on the provider and the model, so check the Model Square and the provider documentation for the current caching terms before you rely on a specific number.
To benefit from caching, keep your repeated content stable and put it at the start of the request. Do not rewrite the system prompt on every call. Do not shuffle the order of large context blocks. Do not append a timestamp or a random ID to the start of the prompt, because that makes the cached content look different and defeats the cache. Small habits like these decide whether caching works or not.
Output tokens cost five times as much as input tokens on Sonnet and Haiku, so long answers are the expensive part of a Claude bill. The fix is to ask for less output.
Be specific about the format you want. Ask for a short answer, a bullet list, or a JSON object instead of an open-ended essay. Set a maximum output length in your code, and treat a response that hits the limit as a signal to split the task, not as a reason to raise the limit. Ask the model to return only the changed code, not the whole file. Ask for a summary, not a rewrite.
Not every Claude call needs an answer in seconds. Summarising documents, labelling data, generating reports, and running nightly evaluations can all wait minutes or hours. If your provider offers a batch mode with a lower price for non-urgent requests, use it for that work.
The idea is simple: you submit a batch of requests, the provider processes them when capacity is available, and you pay less per token than you would for an immediate request. The trade-off is latency. Do not use batch mode for a live chatbot or an interactive coding assistant, where the user is waiting. Do use it for background jobs, where a delay of an hour changes nothing. For more on this, see Batch AI APIs: Processing Millions of Requests Affordably.
Put It All Together with One API Key
All of the techniques above work better when you can switch models without switching providers. If every model needs its own account, its own key, and its own billing, you will not switch models often. You will pick one model and stick with it, and you will pay its price for every request.
A single OpenAI-compatible API key changes that. With Qora API, one key gives you access to Claude, GPT, Gemini, Grok, and 350+ models through one endpoint: https://qoraapi.com/v1. You can route easy requests to Haiku, hard requests to Sonnet, and other work to GPT or Gemini, all from the same code, just by changing the model name.
Here is a minimal example. It calls Claude Sonnet 4.5 through the Qora endpoint. Replace the model name with claude-3-5-haiku-20241022 and the same code calls Haiku instead, at the lower price from the table above.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://qoraapi.com/v1",
api_key=os.environ["QORA_API_KEY"],
)
response = client.chat.completions.create(
model="claude-sonnet-4-5-20250929",
messages=[{"role": "user", "content": "Summarise this in three bullet points."}],
max_tokens=300,
)
print(response.choices[0].message.content)
A note on model names: the exact model name to use is the one shown on the Model Square. If a model name does not work, check the Model Square list first, because names change as new versions are released. The example above uses claude-sonnet-4-5-20250929 as shown for Claude Sonnet 4.5. For a general guide to the OpenAI-compatible setup, see OpenAI-Compatible API: One Key for GPT, Claude & Gemini.
Yes. The cheapest way to use Claude is usually not a different provider, but a cheaper way to call the same models: use Haiku for routine work, cache repeated context, limit output tokens, and batch work that can wait. On Qora, claude-3-5-haiku-20241022 costs $0.15 input and $0.75 output per 1 million tokens, compared with $0.90 and $4.50 for Claude Sonnet 4.5.
Is Claude Haiku good enough for coding?
Haiku is good enough for many coding tasks: small fixes, formatting, explanations, and simple generation. For complex refactoring, long files, or tricky debugging, Sonnet is usually worth the higher price. Test Haiku on your real tasks before deciding, and route only the hard cases to Sonnet.
How much can prompt caching save?
Caching saves money on repeated input, such as a long system prompt or a document you send with every request. The size of the saving depends on the provider, the model, and how stable your input is. Check the current caching terms on the Model Square and the provider documentation before you build a budget around a specific discount.
Should I use Claude Opus or Sonnet?
Use Sonnet as your default for hard work and Opus only when Sonnet fails on a task where failure is costly. Opus is the most capable Claude tier and costs more than Sonnet. This article does not quote an Opus price, because none was verified for this guide. Check the Model Square for the current price.
Can I use one API key for Claude, GPT, and Gemini?
Yes, with Qora API. One key works through the OpenAI-compatible endpoint at https://qoraapi.com/v1 for Claude, GPT, Gemini, Grok, and 350+ models. You switch models by changing the model name in your request, without creating a new account or key for each provider.
Try It
Get $5 free trial credit on request — contact us at [email protected] and we’ll add $5 to your account. Sign up at https://qoraapi.com/, create one API key, and try the same request on Haiku and Sonnet to see the price and quality difference for yourself.
Serverless fits a subset of LLM workloads and fails the rest, and the deciding variable is almost always wall-clock time. Classification, embedding, and short single-shot completions that finish in under ten seconds are cheaper and simpler on a FaaS platform than on a container. Long generations, agent loops, and extended-reasoning calls are not: the platform kills them mid-response, and you pay for tokens the client never received. Decide per endpoint, not per architecture.
The mismatch is structural
The serverless contract assumes work that is short, bursty, stateless, and killable at any instant. LLM calls are long-tailed, incremental, and stateful across turns. Each mismatch maps to a constraint you cannot negotiate away:
Execution time limits. A hard platform ceiling, with no graceful shutdown.
Response size limits. API Gateway caps payloads at 10 MB; Lambda response streaming at 20 MB.
No durable local state./tmp and in-process memory are recycled with the environment.
Cold starts. A fresh environment pays runtime init, dependency loading, and a TLS handshake first.
No guaranteed connection reuse. Pooling happens only if the platform reuses the same warm environment.
At-least-once invocation. Queues and client retries can run your handler twice.
What statelessness costs a chat product
Multi-turn chat needs history, which makes the function stateful by proxy. Pushing it to Redis or Postgres adds a round trip to every turn and turns a pure compute tier into a compute-plus-datastore tier. If that store lives in a VPC, the function’s network path and cold start are coupled to it too. Worse, prompt-prefix caching depends on an exact prefix, so a reordered history invalidates the cache you were counting on.
Platform constraints, compared
Three families matter, and the differences that decide the design are duration, streaming, and connection reuse across requests.
Property
Classic FaaS (AWS Lambda-style)
Edge runtime (Cloudflare Workers-style)
Container platform (Fargate, Cloud Run, Fly.io)
Max duration
15 minutes per invocation; 29 seconds if fronted by API Gateway
No fixed wall-clock limit while a response stream is open; CPU time is capped separately, commonly 30 seconds default and extendable
Cloud Run defaults to 300 seconds per request, configurable to 60 minutes; long-running services have no per-request cap
Streaming support
Only via Function URLs with response streaming; never through API Gateway
Native and first-class; the runtime is built around ReadableStream
Native; the only family that streams reliably through a normal load balancer
Outbound connection reuse
Within a warm execution environment only, and only if the client lives outside the handler
Within an isolate, best-effort; isolates are recycled aggressively
Real keep-alive pools that survive across requests, with tunable pool sizes
Cold start profile
Hundreds of milliseconds to several seconds, driven by runtime, bundle size, and VPC configuration
Low single-digit milliseconds for isolate creation; the real cost is the first outbound TLS handshake in a new isolate
Seconds from zero, but eliminated by keeping a minimum instance count
Runtime limits
Node, Python, Java, .NET, Go, Rust; 128 MB to 10 GB memory; 50 MB zipped deployment package; no inbound sockets
V8 isolates with Web Platform APIs; 128 MB memory; no filesystem; subrequest counts capped per invocation
Whatever fits in a container; no meaningful runtime restrictions
Latency-sensitive short calls and streaming proxies where compute is mostly waiting
Sustained concurrency, long generations, agent loops, in-process prompt caches
The column that quietly decides most architectures is outbound connection reuse. A container keeps a TLS session to the provider open for hours; a function keeps one only while its environment survives, and an edge isolate only until eviction. Duration, memory, and runtime can be worked around. That one cannot, which is why serverless LLM deployments so often route through a long-lived intermediary.
Execution time limits are the design constraint
Treat the platform ceiling as the primary filter. A workable rule of thumb: compare your p95 end-to-end provider latency against roughly a quarter of the platform’s wall-clock budget, because you need headroom for cold start, retries, and the tail. If the budget is 30 seconds, do not inline anything with a p95 above seven or eight seconds. At 15 minutes, the threshold moves to three or four minutes.
Workloads that fit
Short completions. Extraction, rewriting, labelling, short structured output.
Classification and routing. Low single-digit latency, deterministic token counts.
Embeddings. No streaming, bounded output, per-item billing. The cleanest fit here.
Small RAG answers. Retrieve, stuff a bounded context, generate briefly — if retrieval is not a slow VPC hop.
Workloads that do not fit
Long generations. 4,000 tokens at 40 per second is 100 seconds of streaming: fine on an edge runtime, fatal behind a 29-second integration timeout.
Deep reasoning. Time-to-first-token alone can exceed a FaaS front door’s entire budget.
Batch jobs. Per-item overhead dominates; these belong on workers that amortise startup.
Agent loops. A ten-step loop of tool calls multiplies the odds of crossing the ceiling.
A well-behaved serverless LLM handler builds its HTTP client at module scope so the pool survives across warm invocations, sets timeouts per phase, and records when it exceeded its own budget — that log line is the leading signal that the workload is outgrowing the platform.
import os
import time
import httpx
# Module scope. A client created inside the handler pays a fresh TCP and
# TLS handshake on every single invocation.
_client = httpx.Client(
base_url="https://api.qoraapi.com",
timeout=httpx.Timeout(connect=2.0, read=8.0, write=2.0, pool=1.0),
limits=httpx.Limits(max_keepalive_connections=20, max_connections=50),
headers={"authorization": "Bearer " + os.environ["QORA_API_KEY"]},
)
BUDGET_S = 8.0 # the p99 we are willing to absorb inside the function
def classify(text: str) -> str:
started = time.monotonic()
resp = _client.post(
"/v1/chat/completions",
json={
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": "Reply with exactly one label."},
{"role": "user", "content": text},
],
"max_tokens": 8,
"temperature": 0,
},
)
resp.raise_for_status()
elapsed = time.monotonic() - started
if elapsed > BUDGET_S:
# Succeeded, but only because the platform had headroom left.
# Emit this as a metric: it is the leading indicator of a bad fit.
print("over-budget call: %.2fs" % elapsed)
return resp.json()["choices"][0]["message"]["content"].strip()
Streaming from serverless
Streaming exists to move time-to-first-token from seconds to hundreds of milliseconds. Buffering the whole response inside the function destroys that property and adds a failure mode: the entire output must fit in memory and inside the payload limit. If you are going to buffer, you have chosen a synchronous API and should use the job-and-webhook pattern instead.
Passing a stream through a function
The mechanics differ by platform but the shape does not: read the upstream body as a stream, write each chunk to the response stream as it arrives, never accumulate. On Lambda that means response streaming through a Function URL using the streaming handler signature — API Gateway cannot do it, and its payload limit and 29-second integration timeout make the question moot. On edge runtimes it is the default: return a ReadableStream and the runtime forwards chunks as produced.
The heartbeat problem
Every hop between your function and the client has an idle timer, and all of them are shorter than a slow generation. An ALB idles a connection out at 60 seconds by default. nginx’s proxy_read_timeout also defaults to 60. The timer measures inactivity, not total duration, so the fix is to ensure no gap ever approaches it: emit an SSE comment line every 15 seconds. Comment lines begin with a colon and are ignored by every conforming parser, so the cost is a few bytes and the benefit is surviving a model that pauses before its first token.
Two details bite in production. Buffering proxies must be disabled — send X-Accel-Buffering: no and Cache-Control: no-cache, no-transform, or an intermediary holds your chunks until it has enough to flush, reintroducing the latency you removed. And compression must be off; a gzip filter with a 1 KB buffer will sit on your first token until the second arrives.
Cold starts: what actually causes them
Cold start is not one delay. It is four, and they are separable.
Runtime initialisation. Node and Python are fast; a JVM is not.
Dependency loading. A fat SDK that drags in half of AWS adds hundreds of milliseconds.
TLS handshake to the provider. A client built per invocation pays a full handshake before the request is sent.
DNS resolution. Often ignored, occasionally dominant — especially in a VPC, where queries hit the VPC resolver.
Mitigations that work
Smaller bundles. Ship the HTTP client you need, not a framework. Costs nothing at runtime.
Module-scope clients. Build them outside the handler so they survive between invocations.
Provisioned concurrency. Environments already past runtime init and holding an open provider connection. The only fix that removes the handshake cost rather than hiding it.
More memory, on Lambda. CPU scales with memory, so this genuinely shortens CPU-bound init.
What is cargo cult
A scheduled ping every five minutes to keep the function warm does not reliably work. It warms one execution environment, it does not control which environment serves the next real request, and under concurrency you need as many warm environments as simultaneous requests. Warm-up plugins on a schedule share the flaw. The old claim that VPC configuration adds ten seconds to every cold start is obsolete — hyperplane network interfaces fixed it — though VPC still adds latency through NAT hops and DNS.
resource "aws_lambda_function" "llm_proxy" {
function_name = "llm-proxy"
runtime = "nodejs20.x"
handler = "index.handler"
# CPU scales with memory, so this also shortens CPU-bound cold start.
memory_size = 1024
# A hard ceiling. Exceeding it terminates the invocation mid-write.
timeout = 30
filename = data.archive_file.bundle.output_path
source_code_hash = data.archive_file.bundle.output_base64sha256
environment {
variables = {
# An ARN, not a key. The function fetches and caches the secret itself.
QORA_API_KEY_SECRET_ARN = aws_secretsmanager_secret.qora.arn
UPSTREAM_TIMEOUT_MS = "20000"
}
}
# Pre-initialised environments: past runtime init and connection-warm.
# Billed whether or not they serve traffic.
provisioned_concurrency_config {
provisioned_concurrent_executions = 5
}
}
resource "aws_lambda_alias" "live" {
name = "live"
function_name = aws_lambda_function.llm_proxy.function_name
function_version = "$LATEST"
}
# 29000 is the maximum. API Gateway will not let you raise it.
resource "aws_api_gateway_integration" "proxy" {
http_method = "POST"
integration_http_method = "POST"
type = "AWS_PROXY"
timeout_milliseconds = 29000
uri = aws_lambda_function.llm_proxy.invoke_arn
}
Connection reuse and TLS
A TLS handshake to a provider endpoint costs roughly two network round trips: TCP plus TLS negotiation, with TLS 1.3 getting a resumed session down to one. Call it 40 milliseconds to a region a few hundred kilometres away. Against a 20-second generation that is noise. Against a 300-millisecond classification call it is 13 percent of your latency, and if the function also pays runtime init and dependency loading, it can be the second-largest component of the request.
The arithmetic inverts the usual intuition: the shorter your LLM call, the more a cold handshake matters proportionally — which is exactly the workload serverless is supposed to be good at. The fix takes two forms. Where the platform allows it, keep the client at module scope so warm environments reuse the socket; that is free and you should always do it. Where it does not, route through a gateway holding a pooled upstream connection, so your function talks to one nearby endpoint instead of handshaking with a distant provider region. That also removes per-provider pools from your code, and is a large part of why teams put a gateway in front of their providers rather than calling them directly.
Secrets: environment variables are a liability
A provider key in a function’s environment variables is readable by anyone holding lambda:GetFunctionConfiguration, stored in plaintext in your Terraform state, leaked into crash dumps and error payloads, and impossible to rotate without redeploying every function that references it. With forty functions across four environments that is a coordinated fleet-wide deploy, which in practice means the key never rotates.
Secret manager with a cached fetch. Store the ARN, fetch on first use, cache in module scope, refresh on a TTL.
Short-lived credentials. Use the platform identity — IAM role, workload identity, OIDC — to mint a short-lived token.
One scoped key to a gateway. The function holds a single revocable key; the gateway holds the provider credentials. Rotation is one change in one place.
This is the same reasoning behind any sound LLM API security posture: minimise long-lived secrets, and make the ones that remain cheap to rotate.
The cost model: where the crossover is
Serverless wins on spiky traffic and loses on sustained traffic, and the crossover is calculable. Take published on-demand rates: Lambda at $0.0000166667 per GB-second plus $0.20 per million requests, Fargate at $0.04048 per vCPU-hour plus $0.004445 per GB-hour.
Worked example. Each request runs 4 seconds at 512 MB, so it consumes 4 x 0.5 = 2 GB-seconds. Per-request cost is 2 x $0.0000166667 = $0.00003333, plus $0.0000002 for the request itself, totalling $0.00003353.
Spiky, 50,000 requests per day. Lambda: 50,000 x $0.00003353 = $1.68/day, about $50/month. Peak load is near 14 concurrent calls, so containers need two 1 vCPU / 2 GB tasks: 2 x ($0.04048 + 2 x $0.004445) x 730 = $72, plus ~$20 for an ALB. Roughly $92. Serverless wins.
Steady, 1,000,000 requests per day. Lambda: 1,000,000 x $0.00003353 = $33.53/day, about $1,006/month. That averages 11.6 requests per second, roughly 46 concurrent calls; at a conservative 15 per 1 vCPU task, four tasks suffice: 4 x $36.04 = $144, plus $25 for an ALB. Roughly $170. Containers win by about six times.
The crossover sits where Lambda’s monthly bill equals the container floor of about $170: roughly 5.1 million requests per month, or 170,000 per day — a sustained two requests per second. Below that, serverless is cheaper. Above it, you pay a per-invocation premium for elasticity you no longer need.
Profile
Requests per day
Serverless monthly
Container monthly
Cheaper option
Spiky, 4-hour peak window
50,000
~$50
~$92
Serverless
Sustained, flat
1,000,000
~$1,006
~$170
Container, by ~6x
Crossover
~170,000
~$170
~$170
Indifferent on cost; decide on latency and control
Two caveats shift the crossover toward containers. Provisioned concurrency is billed while idle, adding to the serverless column without adding throughput. And if your handler is CPU-bound rather than waiting on the network, the concurrent calls a container task can serve drops sharply.
Failure modes specific to serverless
Silent truncation mid-stream
Once the first SSE chunk is flushed, the HTTP status is committed. If the function then hits its timeout you cannot send a 500 — the client already has a 200 and a partial body. It sees a stream that simply stops, indistinguishable from a network blip, and your application may store a half-finished answer as complete. The fix is an in-band terminal marker: emit an explicit event: truncated frame when the read loop exits without a [DONE] or finish_reason, and make clients treat a stream that closes without one as a failure regardless of status code.
Duplicate execution on retry
Async invocations retry on failure, queues redeliver after a visibility timeout, and API clients retry on connection errors. A retried LLM call is a second billable generation and, if it writes to your database, a duplicated record. You cannot make the provider call idempotent, so idempotency has to live at your boundary: an idempotency key, a short-lived record of completed work, and a client contract that treats a repeat as a lookup. This is the discipline that makes asynchronous job delivery safe, and it applies equally to a synchronous endpoint that retries.
Losing the response when the client disconnects
When the browser tab closes, the function does not necessarily stop. On some runtimes the invocation runs to completion and you pay for every token. On others the runtime aborts the handler, and whether that abort reaches the in-flight upstream request depends entirely on your code. Propagate it explicitly: listen on the request’s abort signal and cancel the upstream fetch, so a disconnected client stops the generation instead of billing you for output nobody reads.
VPC and networking surprises
Placing a function in a VPC to reach a database adds a NAT hop to every outbound provider call: latency plus a per-gigabyte data processing charge. It also changes DNS resolution, so the first lookup in a cold environment can be slower than expected. If the function only needs the public internet and a database with a public endpoint, do not put it in a VPC. If it must, budget the NAT cost into the arithmetic above.
A streaming LLM endpoint on an edge runtime
This handler covers the two cases that break naive implementations: the deadline, and the client walking away. It never buffers, heartbeats on an interval, distinguishes an error before the first byte (where a real status code is still available) from one after (where only an in-band frame is), and aborts upstream on disconnect.
interface Env {
QORA_API_KEY: string;
}
const UPSTREAM = "https://api.qoraapi.com/v1/chat/completions";
const DEADLINE_MS = 90_000; // our own budget, below the platform ceiling
const HEARTBEAT_MS = 15_000; // comfortably under a 60s proxy idle timeout
export default {
async fetch(request: Request, env: Env): Promise<Response> {
if (request.method !== "POST") {
return new Response("method not allowed", { status: 405 });
}
const payload = await request.text();
const upstreamAbort = new AbortController();
// Client disconnect must cancel the upstream generation, not just the response.
request.signal.addEventListener("abort", () => {
upstreamAbort.abort("client-disconnect");
});
const deadline = setTimeout(() => {
upstreamAbort.abort("deadline-exceeded");
}, DEADLINE_MS);
let upstream: Response;
try {
upstream = await fetch(UPSTREAM, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: "Bearer " + env.QORA_API_KEY,
},
body: payload,
signal: upstreamAbort.signal,
});
} catch (err) {
clearTimeout(deadline);
// Nothing has been flushed yet, so a real status code is still available.
return new Response(JSON.stringify({ error: String(err) }), {
status: 504,
headers: { "content-type": "application/json" },
});
}
if (!upstream.ok || upstream.body === null) {
clearTimeout(deadline);
return new Response(upstream.body, { status: upstream.status });
}
const encoder = new TextEncoder();
const decoder = new TextDecoder();
const stream = new ReadableStream<Uint8Array>({
async start(controller) {
const reader = upstream.body!.getReader();
const heartbeat = setInterval(() => {
// SSE comment line: ignored by parsers, resets every idle timer.
controller.enqueue(encoder.encode(": keep-alive\n\n"));
}, HEARTBEAT_MS);
let tail = "";
let sawTerminal = false;
try {
for (;;) {
const { done, value } = await reader.read();
if (done) break;
tail = (tail + decoder.decode(value, { stream: true })).slice(-256);
controller.enqueue(value);
if (tail.includes("[DONE]") || tail.includes('"finish_reason"')) {
sawTerminal = true;
}
}
} catch (err) {
// Status is already committed. The only channel left is in-band.
const frame = JSON.stringify({ message: String(err) });
controller.enqueue(encoder.encode("event: error\ndata: " + frame + "\n\n"));
} finally {
clearInterval(heartbeat);
clearTimeout(deadline);
if (!sawTerminal) {
controller.enqueue(
encoder.encode(
'event: truncated\ndata: {"reason":"upstream ended without a terminal frame"}\n\n'
)
);
}
controller.close();
}
},
cancel(reason) {
// Response stream cancelled downstream: stop paying for the generation.
upstreamAbort.abort(String(reason));
},
});
return new Response(stream, {
headers: {
"content-type": "text/event-stream; charset=utf-8",
"cache-control": "no-cache, no-transform",
"x-accel-buffering": "no",
},
});
},
};
Two things this handler deliberately does not do. It does not retry the upstream call, because a retry after partial output would append a second generation to the first; retries belong before the first byte or in a separate job. And it does not resolve a model to a provider — that is the gateway’s job, and keeping it there means failover across providers happens without redeploying the function.
When not to use serverless, and where I would default
Five conditions disqualify it, and any one of them is sufficient.
Long generations. If p95 time-to-last-token exceeds a quarter of the ceiling, you are one bad tail latency from truncation.
Sustained high concurrency. Past two requests per second you pay an elasticity premium for capacity you could buy outright.
Heavy in-process state. A prompt cache, embedding index, or reranker — anything whose value depends on state surviving between requests.
GPU needs. Cold-starting a GPU costs more than the request you were serving.
A durable provider connection. A long-lived multiplexed connection or websocket upstream rules serverless out.
So: default to serverless for embedding, classification, extraction, short single-shot completions, and the generation half of a small RAG pipeline — anything with a p95 under about eight seconds and bounded output. You get near-zero fixed cost, trivial scaling, and no capacity planning, and the cold-start TLS penalty is manageable with a module-scope client. Default to containers for anything conversational at scale, anything streaming more than a few hundred tokens, anything with an agent loop, and anything where a local cache changes your cost profile. The moment you configure provisioned concurrency to hit a latency SLO, you are paying the container floor while still accepting serverless constraints. Time to move.
The hybrid that actually ships is a thin serverless or edge tier in front of a long-lived worker tier, with a gateway between them holding upstream credentials and pooled connections. The front tier absorbs bursts and terminates client connections; the worker tier owns the slow, stateful, expensive work.
Frequently asked questions
Can I just raise the function timeout?
Up to the platform maximum, yes, and it is often the right first move — Lambda allows 15 minutes per invocation. What you cannot raise is the timeout of the layer in front of it. API Gateway’s 29-second integration timeout is hard, an ALB’s 60-second idle timeout is configurable but bounded, and Cloudflare returns a 524 if an origin does not respond in time. Raising the function timeout only helps if the path bypasses those layers, for example a Function URL with response streaming. Measure the smallest ceiling in the path, not the largest.
Does provisioned concurrency solve cold starts?
It solves the parts you can pre-warm: runtime init, dependency loading, and the TLS handshake if your client lives at module scope. It does not solve anything downstream — provider-side queueing, model warm-up, or your database’s connection establishment. It is also billed whether or not it serves traffic, which makes it the clearest signal that a workload has outgrown serverless.
Can I stream from AWS Lambda?
Yes, but only through a Function URL with response streaming, using the streaming handler signature rather than the standard one. API Gateway cannot stream responses, and its 10 MB payload limit applies to the buffered case. Even with response streaming, plan for the 20 MB soft response limit and for the fact that once you write the first chunk your status code is fixed, so mid-stream failures must be signalled in-band. If streaming is central to your product, an edge runtime fits better than Lambda.
Should I put my provider API key in the function environment?
For a prototype, fine. Past a couple of functions, no. Environment variables are readable by anyone with configuration-read permission, stored in plaintext in your infrastructure state, visible in error payloads, and require a fleet-wide redeploy to rotate. Store the secret’s ARN instead and fetch it once per execution environment with a TTL cache, or issue a single scoped key pointing at a gateway that holds the real provider credentials.
Conclusion
Serverless and edge functions are a good home for short, bounded, stateless LLM calls and a bad home for long, streaming, stateful ones. The decision reduces to wall-clock time against the smallest ceiling in your request path, and it is worth making per endpoint rather than per system. Where the workload fits: keep the HTTP client at module scope, heartbeat your streams, signal truncation in-band, make retries idempotent, and never put a long-lived provider key in an environment variable. Where it does not: move to a container and stop paying for elasticity you are not using. For the middle ground — bursty front ends, pooled upstream connections, one place to rotate credentials — a gateway such as QoraAPI lets a short-lived function behave like a long-lived client.
Validate the model’s output because you cannot prove the absence of a failure mode by writing a better instruction. A prompt is a request, not a contract. The same input produces different output across model versions, across providers serving the same weights, and across supposedly identical temperature-0 calls, because batching, kernel selection and floating-point reduction order are not deterministic. Output guardrails are the layer that turns “the model usually does this” into “the system only ever ships this” — and they run after generation, not inside the prompt.
Why you validate the output instead of hardening the prompt
Instruction-following is best-effort. Writing “always return valid JSON” shifts the probability distribution of outputs; it does not create an invariant. That matters because prompt failures are silent: a model that ignores a format instruction returns something plausible until a parser hits it at 3am, whereas a validator fails loudly, at the boundary, with the offending value attached.
Non-determinism has several sources and only one is the temperature parameter. Greedy decoding at temperature 0 breaks ties in the logits, but the logits are a floating-point computation whose reduction order depends on batch size, sequence packing and tensor-parallel layout. Change the batch size and a token ahead by 0.001 logits can flip. Mixture-of-experts routing adds another: a request landing on a different expert set produces a different continuation from the same checkpoint. Version pinning narrows the distribution rather than freezing it, because a pinned snapshot can still be served on different hardware.
So a prompt regression suite tells you what changed, not what is safe right now. Prompts need versioning and review; guardrails are runtime code that needs tests, metrics and an owner. This is about validating what the model says, not what it does — constraining actions is a separate control with a separate threat model, covered in sandboxing AI tool calls.
The validation layers, cheapest first
Run the layers in ascending cost order and stop at the first blocking verdict. A structural failure makes every later check meaningless, and a regex hit is cheaper to act on than a classifier score. Ordering also keeps expensive layers’ error rates out of the picture when a cheap layer already has the answer.
Near zero if the schema is derived from real payloads; high if the schema is aspirational
Deterministic rules (regex, allow-lists, bounds)
under 1 ms
CPU only
Secret-shaped strings, banned phrases, competitor names, over-length answers, numbers outside a plausible range, tool names not on the allow-list
Entirely a function of how precisely the patterns are written; over-broad regex is the largest source of false positives in most stacks
Groundedness (claim vs retrieved context)
0.3-2 s with a judge; under 50 ms for the prefilter
Cheap for substring and embedding lookups, one model call for the residue
Claims with no support in the context, and claims the context directly contradicts
High when retrieval was truncated or the answer legitimately draws on parametric knowledge; needs a prefilter or it blocks good answers
Classifier-based policy checks
20-150 ms
GPU or hosted classification endpoint, priced per request
Toxicity, system-prompt self-disclosure, injection payloads echoed back, competitor mentions, PII in free text
Dominated by the threshold you pick; you cannot reason about it, you have to measure it on labelled data
Human review (sampled)
Minutes to hours
Highest, by orders of magnitude
Novel failure modes, calibration of the automated layers, cases where two classifiers disagree
Not applicable, but throughput is capped at tens to hundreds of items per reviewer per day
Two properties matter more than the list. Each layer should return findings rather than a boolean, so the final verdict comes from a policy object instead of being hard-coded inside each check. And the false-positive column decides whether a layer ships, not the true-positive column: a detector that catches every real leak but blocks 8 percent of valid traffic is not a detector, it is an outage.
Schema validation in practice
Make the schema strict enough to be worth running
A schema asserting {"type": "object"} catches nothing. The useful constraints are the boring ones: additionalProperties: false so invented fields surface instead of passing downstream, required on every field the consumer dereferences, enum on anything categorical, and maxItems and maxLength to bound payload size.
Provider strict modes are worth enabling — the strict json_schema response format, Anthropic tool-use schemas, Gemini’s responseSchema — but they remove syntax-level failures, not semantic ones, and they do not survive a failover. A strict schema guarantees the shape of the answer, never its truth. Mechanics are in structured outputs and JSON mode.
A repair loop with a hard attempt cap
When validation fails, the cheapest fix is usually to re-ask the same model with the validator’s error attached. That works often, because the failure is frequently a formatting slip rather than a reasoning failure. It needs a cap: every attempt is a full round trip with full input tokens, so two attempts on a 4,000-token prompt is three times the input cost of a single call.
import json
from dataclasses import dataclass
from typing import Any, Callable
import jsonschema
INVOICE_SCHEMA: dict[str, Any] = {
"type": "object",
"additionalProperties": False,
"required": ["vendor", "currency", "total_cents", "line_items"],
"properties": {
"vendor": {"type": "string", "minLength": 1, "maxLength": 200},
"currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
"total_cents": {"type": "integer", "minimum": 0},
"line_items": {
"type": "array",
"minItems": 1,
"maxItems": 200,
"items": {
"type": "object",
"additionalProperties": False,
"required": ["description", "amount_cents"],
"properties": {
"description": {"type": "string", "maxLength": 500},
"amount_cents": {"type": "integer", "minimum": 0},
},
},
},
},
}
MAX_REPAIR_ATTEMPTS = 2
@dataclass(frozen=True)
class SchemaResult:
ok: bool
value: dict[str, Any] | None
attempts: int
errors: list[str]
def _describe(exc: Exception) -> str:
if isinstance(exc, jsonschema.ValidationError):
path = "/".join(str(p) for p in exc.absolute_path) or "<root>"
return f"{path}: {exc.message}"
return str(exc)
def parse_with_repair(
raw: str,
call_model: Callable[[str, float], str],
temperature: float = 0.0,
) -> SchemaResult:
"""Validate raw output; on failure re-ask with the validator error attached.
call_model is injected so this is unit-testable without a network call.
"""
errors: list[str] = []
candidate = raw
for attempt in range(MAX_REPAIR_ATTEMPTS + 1):
try:
value = json.loads(candidate)
jsonschema.validate(value, INVOICE_SCHEMA)
except (json.JSONDecodeError, jsonschema.ValidationError) as exc:
errors.append(_describe(exc))
else:
return SchemaResult(True, value, attempt, [])
if attempt == MAX_REPAIR_ATTEMPTS:
break
candidate = call_model(
"Your previous reply failed validation. Fix only the problems listed. "
"Do not add, remove or reinterpret fields, and do not change a value "
"that was not listed as invalid.\n"
"Errors:\n" + "\n".join(f"- {e}" for e in errors[-3:]) + "\n"
"Return the corrected JSON object and nothing else.",
temperature,
)
return SchemaResult(False, None, MAX_REPAIR_ATTEMPTS, errors)
What the repair loop must never do
The repair prompt above carries a sentence doing real work: do not change a value that was not listed as invalid. Without it, a model asked to fix a schema error will often rewrite the data to make the error disappear — rounding a total so the line items sum, deleting the field with the wrong type, reclassifying a currency to fit the enum. That is data corruption dressed as a successful repair.
Cross-field semantic checks belong outside the repair loop. If your validator asserts that sum(line_items) == total_cents, a repair cannot fix a mismatch, because the model cannot know which side is wrong. Block and route to a human. A repair rate jumping from 2 percent to 15 percent after a prompt edit is the earliest signal of a regression.
Groundedness: unsupported is not the same as contradicted
Extract the atomic claims, then look for support for each in the context you actually retrieved. Three outcomes, and the third is the important one. Supported: a span in the context entails the claim. Unsupported: nothing speaks to it either way — the model may be using parametric knowledge, or retrieval was truncated. Contradicted: the context asserts the opposite. Contradicted claims should be blocked or rewritten; unsupported claims should be downgraded or shown with a citation requirement, because blocking every unsupported claim destroys a summarizer that adds one reasonable inference. See detecting and reducing hallucinations for the taxonomy in depth.
Per-claim judging is quadratic in the obvious implementation, so add a prefilter: normalize each claim, then check for a near-verbatim span, an embedding similarity above a tuned threshold, or a token-overlap ratio. Only the residue goes to a judge, and the judge call is batched — one request containing every unresolved claim, not one per claim.
Worked example. A response contains 40 atomic claims. The prefilter resolves 28 of them, or 70 percent, without a model call. The remaining 12 go to a judge in one batched request returning per-claim entailment labels in roughly 400 ms. Added latency is one round trip, not twelve. Calling the judge once per unresolved claim at 350 ms each would add 4.2 seconds, and the feature would be disabled within a week.
If you use a judge, use a different model than the generator, or at minimum a different prompt with a strict rubric. Ask for a span-level verdict — “quote the sentence that supports this claim, or answer NONE” — rather than a scalar score. A judge forced to produce a citation cannot reward itself with a vague 0.8.
PII and secret leakage in the output
Deterministic detectors cover more than people expect: card numbers with a Luhn check, IBANs with mod-97, national identifier formats, AWS access key prefixes, JWTs by their three-segment structure, PEM private key blocks, connection strings with embedded credentials. Use the checksum wherever the format has one, because a bare digit pattern also matches order numbers.
Regex alone misses the cases that matter: a paraphrased address, a phone number written in words, a name split across a token boundary, an email base64-encoded into a code block, “the card ending in 4242”, or an identifier the model reconstructed from context rather than copied. For free-text names, addresses and locations you need a NER or classifier pass in addition, and it will be the layer with the least predictable error rate.
Redaction versus blocking is a decision about reversibility, not severity. Mask an email as a stable placeholder when the consumer only needs to know an email was present, and keep the mapping server-side. Block when the value is a credential, a regulated identifier, or anything whose exposure is itself the harm. And never log the raw pre-guardrail output: if your guardrail redacts PII and your observability pipeline then stores the original response body, the guardrail has achieved nothing except latency. Log the post-guardrail text plus finding metadata — layer, rule, span offsets, verdict — and put raw output, if you must keep it, in a separate store with short retention and its own audit trail. The wider constraints are in AI data privacy and GDPR.
Fail-open or fail-closed, decided per surface
Fail-open means that when the guardrail itself errors or times out, the output ships. Fail-closed means it does not. The answer is not global, and it is not something you discover in an except block — it is configuration attached to the surface.
Blast radius decides. A support chatbot failing open ships one bad answer to one user who can ask again; the damage is bounded. The same chatbot failing closed turns every classifier timeout into “something went wrong” for the whole user base, an availability incident you caused yourself. A clinical triage flow inverts the calculus: failing open can produce advice that causes physical harm, while failing closed sends the user to a phone number a human answers. Internal analytics fails open by default, because a wrong number in a dashboard is cheap to correct.
Be precise about timeouts. A classifier that times out has said nothing about the content; treating that as “unsafe” produces random outages under load, and load is exactly when timeouts cluster.
Latency budgeting and the streaming problem
Guardrails add round trips, and the budget is tighter than teams assume. Run deterministic checks inline; they cost microseconds. Run the classifier and the groundedness check concurrently, because they are independent, so the wall-clock cost is the maximum rather than the sum.
Worked example. A response is 600 tokens. The PII detector takes 15 ms, the toxicity classifier 90 ms, the policy classifier 120 ms and the groundedness judge 400 ms. Serially that is 625 ms of added latency before the first token reaches the client. With asyncio.gather over the last three it is max(90, 120, 400) = 400 ms. The judge is the critical path, which is why the groundedness prefilter buys more than any micro-optimization of the regexes.
Hold-back windows and retraction UX
Streaming and blocking are in direct tension. Once a token is on the user’s screen you cannot un-show it; you can only append a retraction, and you should assume the user read the original. Three patterns work. First-N-token gating buffers the opening 40 to 80 tokens, validates it, and releases the stream if it passes; most violations are visible in the opening. A sliding hold-back window keeps the last W tokens unreleased while checks run, bounding exposure to the window. Full buffering generates, validates, then renders, which for a 600-token answer at 60 tokens per second moves time-to-first-token from about 0.5 s to about 10 s.
type GateResult = { block: boolean; reason?: string };
export type Gate = {
holdbackChars: number;
check: (unreleased: string) => Promise<GateResult>;
};
export async function* guardedStream(
source: AsyncIterable<string>,
gate: Gate,
): AsyncGenerator<string> {
let pending = "";
let released = 0;
for await (const chunk of source) {
pending += chunk;
// Only inspect text that has not reached the client yet.
const unreleased = pending.slice(released);
if (unreleased.length < gate.holdbackChars) continue;
const { block, reason } = await gate.check(unreleased);
if (block) {
yield `\n\n[Response withheld: ${reason ?? "policy"}]`;
return;
}
yield unreleased;
released = pending.length;
}
const tail = pending.slice(released);
if (tail) {
const { block, reason } = await gate.check(tail);
yield block ? `\n\n[Response withheld: ${reason ?? "policy"}]` : tail;
}
}
The window bounds the text a user can see before a violation is caught, so a retraction undoes a sentence rather than a paragraph. It does not help when the violation is a single token in the middle of a long answer. For high-harm surfaces, accept the latency and buffer the whole response.
Fallback strategies
A blocked response is a routing decision, not an error state. Choose the fallback before you ship the guardrail, because inventing one during an incident produces worse outcomes than a boring canned string.
Fallback
Use when
Added latency
Cost per event
What the user sees
Safe canned response
The block was for an out-of-scope or off-policy request
Under 10 ms
Effectively zero
A short answer that declines and points to docs or a human
Degrade to a smaller model with a stricter prompt
The violation looks prompt-shaped and a constrained model is good enough for the task
One extra generation, 300 ms to 2 s
One additional generation plus the blocked one
A slightly less rich but valid answer
Hand off to a human
The domain is high-harm and the request is legitimate but unresolvable automatically
Minutes
Highest, and unbounded per item
A queue position, a callback promise or a ticket number
Structured error
The caller is a machine, not a person
Under 10 ms
Effectively zero
A typed error with a retryable flag and the failing rule id
Two rules of thumb. Degrade to a smaller model when the failure came from a permissive prompt, and not when the task is simply hard — a weaker model on a hard task produces a confident wrong answer, which is worse than a block. And never let a fallback re-enter the same guardrail loop without a depth limit, or a persistent violation becomes an infinite regeneration loop.
Measuring a guardrail you intend to ship
Build a labelled set before shipping the layer: a sample of real traffic, human-labelled at the exact verdict granularity you ship, plus a synthetic adversarial set covering the failures you worry about. Then track precision, recall and the false-positive rate on live traffic, split by layer, because a blended number hides which layer is misfiring.
Worked example. You serve 1,000,000 responses a month and the blocking guardrail fires on 0.5 percent of them, so 5,000 blocks. If the measured false-positive rate on known-good outputs is 5 percent, then roughly 0.05 x 995,000 = 49,750 legitimate responses were blocked. You have manufactured an incident an order of magnitude larger than the one you were preventing. You cannot measure that 5 percent from blocked traffic alone — it requires labelling outputs you allowed, which is the step teams skip.
Two more metrics belong on the same dashboard: repair-loop rate, a leading indicator of prompt or model regression, and block rate per rule, because a rule whose block rate is zero is either perfect or broken and it is almost never perfect. Re-run the labelled set on every model version change. Keeping model and prompt changes traceable alongside these numbers is where LLM observability earns its keep. A guardrail nobody measured is one you cannot ship.
A concrete pipeline
The shape that survives production is a set of layer functions returning findings, a policy object resolved per surface, and a verdict computed by severity rather than control flow. The classifier is injected, so the whole thing is testable with a stub.
import asyncio
import re
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Awaitable, Callable
class Verdict(str, Enum):
ALLOW = "allow"
REDACT = "redact"
BLOCK = "block"
REVIEW = "review"
@dataclass
class Finding:
layer: str
rule: str
verdict: Verdict
detail: str
@dataclass
class Outcome:
verdict: Verdict
text: str
findings: list[Finding] = field(default_factory=list)
timings_ms: dict[str, float] = field(default_factory=dict)
SECRET_PATTERNS = {
"aws_access_key": re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b"),
"private_key_block": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
"jwt": re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b"),
}
Policy = dict[str, Verdict]
# One explicit policy per surface. Fail-open and fail-closed are data here,
# not an except block somewhere in the request handler.
POLICIES: dict[str, Policy] = {
"support_chat": {
"structural": Verdict.BLOCK, "policy": Verdict.BLOCK,
"grounded": Verdict.REDACT, "on_error": Verdict.ALLOW,
},
"clinical_triage": {
"structural": Verdict.BLOCK, "policy": Verdict.BLOCK,
"grounded": Verdict.BLOCK, "on_error": Verdict.BLOCK,
},
"internal_analytics": {
"structural": Verdict.REVIEW, "policy": Verdict.REVIEW,
"grounded": Verdict.ALLOW, "on_error": Verdict.ALLOW,
},
}
WORST_FIRST = (Verdict.BLOCK, Verdict.REDACT, Verdict.REVIEW, Verdict.ALLOW)
def _worst(findings: list[Finding], fallback: Verdict) -> Verdict:
for verdict in WORST_FIRST:
if any(f.verdict is verdict for f in findings):
return verdict
return fallback
def redact(text: str, findings: list[Finding]) -> str:
out = text
for f in findings:
if f.layer == "pii" and f.detail:
out = out.replace(f.detail, f"[REDACTED:{f.rule}]")
return out
async def run_guardrails(
text: str,
surface: str,
*,
validate_structure: Callable[[str], list[Finding]],
classify: Callable[[str], Awaitable[list[Finding]]],
check_groundedness: Callable[[str], Awaitable[list[Finding]]],
) -> Outcome:
policy = POLICIES[surface]
timings: dict[str, float] = {}
start = time.perf_counter()
structural = validate_structure(text)
timings["structural"] = (time.perf_counter() - start) * 1000
if structural:
# A structural failure makes every later layer meaningless.
return Outcome(_worst(structural, policy["structural"]), text, structural, timings)
start = time.perf_counter()
rule_findings = [
Finding("rules", name, Verdict.BLOCK, "secret-shaped string in output")
for name, pattern in SECRET_PATTERNS.items()
if pattern.search(text)
]
timings["rules"] = (time.perf_counter() - start) * 1000
start = time.perf_counter()
policy_findings, grounded_findings = await asyncio.gather(
classify(text), check_groundedness(text)
)
timings["classifier_and_grounded"] = (time.perf_counter() - start) * 1000
findings = rule_findings + policy_findings + grounded_findings
verdict = _worst(findings, Verdict.ALLOW)
if verdict is Verdict.REDACT:
text = redact(text, findings)
return Outcome(verdict, text, findings, timings)
Three choices in that code are load-bearing. Structural failures short-circuit, because there is nothing useful to say about the policy of unparseable output. Classifier and groundedness run under one gather, so added latency tracks the slower of the two. And the error path lives in POLICIES rather than a bare except, so an operator can change it without reading the request handler.
What I would ship first for a customer-facing product
Ship structural validation with a bounded repair loop first, before any classifier. It is free, its false-positive rate is near zero when the schema comes from real payloads, and it eliminates the failures that page you at 3am: truncated JSON, missing fields, wrong types. Instrument the repair rate from day one.
Second, ship deterministic PII and secret detection, fail-closed on secrets and redact on everything else, with raw output kept out of logs from the first commit. It is cheap, defensible in a security review, and the layer most likely to catch a genuine incident.
Third, buffer the full response for the single highest-harm surface, and only that surface. Non-streaming is a real product cost, so spend it where the blast radius justifies it.
Do not ship a policy classifier as a blocker before you have a labelled set and a measured false-positive rate. Do not ship an LLM-as-judge groundedness check as a blocker at all in the first release: run it in shadow mode, log what it would have blocked, and label a few hundred of those decisions by hand. Until you can state your false-positive rate with a number attached, it is a measurement, not a control.
Put model routing behind a gateway rather than wiring providers into application code, so pinning a version, swapping a judge or failing over is a configuration change instead of a deploy.
Frequently asked questions
Do I still need output validation if I use a provider’s strict structured outputs mode?
Yes, for three reasons. Strict modes constrain syntax, not semantics: a strict schema will happily return a well-typed total that does not equal the sum of its line items. They are provider-specific, so the guarantee disappears the moment your gateway fails over. And they say nothing about groundedness, PII or policy.
Should the groundedness judge be a different model from the generator?
Preferably yes, and at minimum it must be a different prompt with an explicit rubric. A judge sharing the generator’s system prompt inherits the same framing and will rationalise the generator’s output. If you can only afford one model, demand a quoted supporting span for every supported verdict.
Can I stream tokens and still enforce guardrails?
Partially. Use a hold-back window so the text you validate has not reached the client, and accept that a late violation still needs a retraction. First-N-token gating covers violations in the opening sentence. Where a leaked token is unacceptable, do not stream.
How do I handle a guardrail false positive in production?
Make it reviewable rather than just blockable. Store the finding, the rule id, the span and a hash of the output, so a reviewer can judge the decision without storing raw text, and give the reviewer an override that feeds back into the labelled set. If one rule accounts for most of your false positives, fix that rule rather than raising the layer threshold.
Do guardrails belong in the client, the gateway, or the application?
Split them by what each knows. The application owns semantic checks, because only it knows the schema, the retrieved context and the policy per surface. The gateway owns the cross-cutting pieces: version pinning, provider failover, redacted logging, rate limits. The client should own nothing that matters for safety.
Conclusion
Output guardrails are the executable specification of what your system is allowed to emit. Order them by cost, make each layer return findings rather than booleans, resolve fail-open versus fail-closed per surface as configuration, and measure the false-positive rate before anything is allowed to block. The layers that ship first are the boring ones: schema validation with a bounded repair loop, deterministic secret and PII detection, and full buffering where a leaked token is unacceptable.
The alternative is not “no guardrails”, it is unmeasured guardrails: regexes nobody tuned, a threshold copied from a blog post, and no idea how many good answers you blocked last month. Routing model traffic through a single OpenAI-compatible endpoint such as qoraapi.com makes the operational side tractable — pinned versions, failover, one place to enforce redaction and logging — but the validation logic is still yours to design.
