An LLM API bill is priced per token, not per request and not per word: two rates for input and output, plus a third cheaper rate for input the provider has already seen. Caching, batching, model choice and reasoning effort are all ways of moving tokens between those three buckets.
Every rate below is illustrative, chosen so the arithmetic stays visible. Real rates differ between providers and change.
The billing unit is the token
A token is a subword unit produced by the model’s tokenizer. For clean English prose, four characters per token is the usual rule of thumb, which puts roughly 750 words at roughly 1,000 tokens — fine as a sanity check, dangerous as a budget input.
Structured content tokenises worse. Code lands nearer three characters per token. Base64 blobs, hex digests, UUIDs and long digit strings are close to the worst case, because the tokenizer has no vocabulary for them and falls back to fragments. CJK text is a different regime: one Chinese character is often one token, so a document a quarter the length of an English one costs the same.
| Content | Approximate characters per token | What drives the cost |
|---|---|---|
| English prose | 4 | Common subwords are single tokens |
| Source code | 3 | Short identifiers, punctuation, indentation |
| JSON and YAML | 2.5 to 3 | Every key name, quote, brace and comma is billed |
| Base64, hex, UUIDs | 1.5 to 2 | No vocabulary; tokenizer emits fragments |
| CJK text | 1 to 1.5 | Whole characters are single tokens |
| Long digit strings | 1.5 to 2 | Digit grouping is inconsistent across tokenizers |
JSON pays for schema, not information: twenty keys averaging ten characters spend roughly 250 characters, on the order of 90 tokens, on key names, quotes and colons before a single value is transmitted. An estimator that splits on whitespace sees a minified JSON blob as one word and is wrong by an order of magnitude.
Input tokens versus output tokens
Input is everything the model reads; output is everything it writes, and output typically costs several times more per token.
The reason is mechanical. Prompt processing is a single forward pass over a known sequence, so the work parallelises across the accelerator’s compute units. Generation is sequential by construction: token n+1 cannot be computed until token n exists, so the model runs one step at a time, each step reading the entire growing key-value state. That workload is memory-bandwidth-bound rather than compute-bound, so cost per token is much higher.
| Illustrative tier | Input USD / million | Cached input USD / million | Cache write USD / million | Output USD / million | Batch output USD / million |
|---|---|---|---|---|---|
| Small and fast | 0.25 | 0.025 | 0.31 | 1.25 | 0.625 |
| Mid-tier general | 3.00 | 0.30 | 3.75 | 15.00 | 7.50 |
| Frontier reasoning | 15.00 | 1.50 | 18.75 | 75.00 | 37.50 |
Output is the expensive bucket per token; input is the large bucket by volume. In the worked example below, input outnumbers output roughly 18 to 1 in tokens, turning a fivefold price ratio into 3.6-fold in dollars. Tuning the completion while ignoring the prompt optimises the smaller half of the bill.
The complete token bill for one request
All of the following is input, and all of it is re-billed on every call:
- The system prompt: persona, policy and output-format rules.
- Tool schemas: descriptions, parameter names, enums.
- Retrieved context: RAG chunks, search results, file excerpts.
- The full conversation history so far.
- The current user message.
Output is the visible completion plus, on reasoning models, the internal tokens generated before the answer begins.
Multi-turn chat grows quadratically
A stateless completion API keeps no session state, so a chat client re-sends the entire history every turn. Cost per turn grows linearly with turn number, so total cost grows with the square of conversation length. For a ten-turn conversation with a 2,500-token system and tool block, 150-token user messages and 350-token replies:
- Input on turn 1: 2,500 + 150 = 2,650 tokens.
- Input on turn 10: 2,500 + (9 x 500) + 150 = 7,150 tokens.
- Total input: (10 x 2,650) + 500 x (0 + 1 + … + 9) = 26,500 + 22,500 = 49,000 tokens.
- Output: 10 x 350 = 3,500 tokens.
- Unique content is only 2,500 + 1,500 + 3,500 = 7,500 tokens, so you paid 6.5 times over for the privilege of being stateless.
At the illustrative mid-tier rates that is 49,000 x 3.00 / 1,000,000 = 0.147 USD of input plus 3,500 x 15.00 / 1,000,000 = 0.0525 USD of output, about 0.20 USD. Twenty turns cost 148,000 input tokens — three times the total for twice the turns.
Cached input: what a prompt cache actually caches
A prompt cache stores the model’s internal key-value state for a prefix of the prompt. If a later request has a byte-identical prefix in the same position, the provider skips the prefill computation for those tokens and bills them at the cached-input rate: typically 80 to 90 percent below fresh input, plus a small surcharge on the write that populates the cache. Two conditions decide whether it fires, both of them design decisions.
- The prefix must be byte-identical. One character different — a timestamp, a user name, a reordered tool list — and every request misses. Serialising JSON dictionaries without sorting keys is enough to break it.
- The shared prefix must come first. Caches match forward from the start of the prompt, so anything variable placed before the static block invalidates everything after it.
The ordering mistake that quietly costs a quarter of the bill
The natural way to write a template is to put the per-user instruction where a human would read it, at the top. So teams build f"You are assisting {user_name} at {company}. " + SHARED_SYSTEM_PROMPT and then wonder why the cache never hits. Move every variable — user name, tenant, locale, current date, retrieved documents — after the static system prompt and tool schemas. The template becomes less readable and the bill drops.
Retrieved context is variable by definition and can never be in the cached prefix, so order multiple retrievals most-stable-first. TTL is the other lever: caches expire after a few minutes of inactivity, and some providers sell a longer window at a higher write cost. Continuous traffic keeps a short TTL warm for almost no write overhead; bursty traffic means paying the write repeatedly for prefixes nobody reuses. Treat the prefix as a versioned interface with a monitored hit ratio: a template change that drops that ratio from 0.95 to zero is a large cost increase no deploy diff will show you. The prompt caching guide covers prefix construction.
Reasoning tokens are output tokens
Models that reason before answering generate internal tokens billed at the output rate even though they never appear in the returned text. Some APIs report them separately as reasoning_tokens, others fold them into the completion count, and a few never surface them.
Output length therefore stops being a property of your prompt and becomes a property of the problem. A request producing a 400-token answer can generate 4,000 reasoning tokens on a hard input: at the illustrative mid-tier rate, 4,000 x 15.00 / 1,000,000 = 0.06 USD of invisible output against 0.006 USD of visible output.
This is how a cheap model becomes expensive. A small model at 1.25 USD per million output tokens is genuinely a tenth of the price per token. But if it fails to converge and burns 30,000 reasoning tokens, that call costs 0.0375 USD, while a stronger model settling in 2,000 tokens costs 0.03 USD. The cheap model lost on price and produced the worse answer. Cap the effort or thinking-budget setting where a provider offers one, log reasoning tokens per request, and alert on the p95 rather than the mean. The reasoning models guide covers picking an effort level per task class.
Batch APIs trade latency for price
Asynchronous batch endpoints accept a file of requests and return results within a turnaround window, commonly up to 24 hours, in exchange for a substantial discount on both input and output — often around half. If nothing user-facing is waiting, batch is free money; if something is waiting, it is not a discount at all. Nightly summarisation, embedding backfills, eval scoring, bulk classification and moderation sweeps qualify; anything inside a request path does not.
Three failure modes to design around. Results expire, so a job read a day after it completes may return nothing. Failures arrive per request rather than as a job-level error, so you need per-line reconciliation and a resubmission path. And a job that fails validation on submission still consumed the upload, so validate the JSONL locally first. The batch processing guide walks through the job lifecycle.
Non-text modalities are metered on different units
Images are priced per image with the price derived from resolution — by tiling or by normalising to a pixel budget — so a 4K screenshot can cost more than several pages of text. Audio input is priced per second of audio and silence is billed; audio output is priced per character or per second. Video is priced per second or by sampled frames billed at the per-image rate, so a 60-second clip can exceed the entire text budget around it.
| Modality | Metering unit | Relative magnitude | What surprises people |
|---|---|---|---|
| Image input | Per image, derived from resolution or tiles | Comparable to a long text prompt | Downscaling often costs no accuracy and cuts this substantially |
| Image generation | Per image, by size and quality | Orders above text generation | Size and quality settings multiply, they do not add |
| Audio input | Per second of audio | Cheap per minute | Silence and hold music are billed |
| Audio output | Per character or per second | Cheap per thousand characters | Long-form narration dominates the request |
| Video input | Per second, or per sampled frame | Usually the most expensive unit here | Frames multiply the per-image rate by frame count |
| Embeddings | Per input token | An order of magnitude below generation | Every re-index pays again for the whole corpus |
Log the modality breakdown separately. A single multimodal request can dominate a monthly bill while looking like one request in your metrics.
The cost multipliers nobody models
- Retries. A retry bills twice, and if the first attempt failed after prompt processing you paid input for output you never received. A 3 percent retry rate adds 3 percent; a retry that resends an enlarged context adds far more.
- Guardrail-triggered regeneration. If 4 percent of completions fail validation and are regenerated, output cost rises 4 percent — plus the validator, which for an LLM-as-judge check is a second inference per request.
- System prompt duplication. A 2,500-token system prompt on 600,000 requests a month is 1,500 million input tokens, or 4,500 USD at the illustrative mid-tier rate. It is often the largest single line item and it is invisible in any dashboard reporting per-request cost.
- Verbose tool schemas. Twenty tools at 250 tokens each is 5,000 tokens per call, paid on every turn including turns that invoke no tool.
- Side costs. Embedding every chunk for a vector store, re-embedding on each re-index, and observability platforms billing per span or stored event.
Worked example: a support chatbot
Assume a static system prompt of 1,800 tokens, static tool schemas of 700, retrieved context of 2,500, average history of 1,200 and a user message of 150. Input is 1,800 + 700 + 2,500 + 1,200 + 150 = 6,350 tokens. Output is 350 tokens with no reasoning. Volume is 20,000 requests per day, 600,000 per month. Rates: 3.00 USD per million input, 15.00 per million output, 0.30 per million cached input, 3.75 per million cache write.
Baseline. Input: 600,000 x 6,350 = 3,810 million tokens at 3.00 USD = 11,430 USD. Output: 600,000 x 350 = 210 million tokens at 15.00 USD = 3,150 USD. Total 14,580 USD per month.
With prompt caching. The cacheable prefix is the system prompt plus tool schemas: 2,500 tokens, byte-identical, positioned first. Assume a five-minute TTL and continuous traffic, so essentially every request hits. Cached tokens: 600,000 x 2,500 = 1,500 million at 0.30 USD = 450 USD, against 4,500 USD uncached — a saving of 4,050 USD. Cache writes: 288 TTL windows per day x 2,500 tokens = 21.6 million tokens per month at 3.75 USD = 81 USD. New total: 11,430 – 4,050 + 81 + 3,150 = 10,611 USD, a 27 percent reduction. At a 0.90 hit ratio, add back 150 million tokens at 3.00 USD = 450 USD, for about 11,061 USD.
Adding batch for the offline slice. Suppose 30 percent of volume is a nightly transcript-scoring job with no user waiting: 180,000 requests per month, routed to the batch endpoint at a 50 percent discount on uncached input and output. Those requests still read the cached prefix at 135 USD. Their uncached input is 180,000 x 3,850 = 693 million tokens at 1.50 USD = 1,039.50 USD, and their output is 180,000 x 350 = 63 million tokens at 7.50 USD = 472.50 USD, for a slice total of 1,647 USD against 3,159 USD run synchronously. The remaining 420,000 online requests cost 315 + 4,851 + 2,205 = 7,371 USD. Grand total: 9,099 USD per month, 37.6 percent below baseline.
| Scenario | Uncached input USD | Cached input USD | Cache writes USD | Output USD | Monthly total USD |
|---|---|---|---|---|---|
| Baseline | 11,430.00 | 0.00 | 0.00 | 3,150.00 | 14,580.00 |
| Prompt caching on the static prefix | 6,930.00 | 450.00 | 81.00 | 3,150.00 | 10,611.00 |
| Caching plus batch for 30 percent of volume | 5,890.50 | 450.00 | 81.00 | 2,677.50 | 9,099.00 |
Forecasting before you launch
The formula: requests per day times 30, times the sum of uncached input tokens over a million times the input rate, cached prefix tokens over a million times the cached rate, and output tokens over a million times the output rate, plus side costs.
Two numbers dominate the uncertainty and neither is the price. The first is tokens per request, which in a chat product is not a constant: it grows with conversation length, retrieval depth and tool-catalogue size, and a p50-to-p95 spread of three times is normal. The second is requests per day, easy to underestimate for anything with fan-out — an agent that loops, or a nightly job that scales with the size of a customer’s data rather than the number of customers.
Price changes are the third-order risk: public, announced, and applying to everyone. Instrument real token counts from the first request, split by feature and tenant, and reconcile them against the invoice monthly — the gap between what your logs say and what you were billed for is where retries, failed attempts and side costs hide. The usage metering and billing pipeline makes that reconciliation possible, and LLM observability is where per-request counts should already be flowing. A gateway reporting usage across providers behind one key gives you the minimum required, which is the role qoraapi.com plays.
Counting tokens and projecting cost in Python
The first script counts real tokens and compares a JSON serialisation against the same facts as prose.
import json
import tiktoken
ENC = tiktoken.get_encoding("o200k_base")
def n_tokens(text: str) -> int:
return len(ENC.encode(text, disallowed_special=()))
system_prompt = open("prompts/support_v7.md", encoding="utf-8").read()
tool_schemas = json.dumps(json.load(open("tools/support_tools.json", encoding="utf-8")))
def input_tokens(history, retrieved_docs, user_message):
"""Everything the model reads is input. Miss one part and the forecast is wrong."""
parts = [system_prompt, tool_schemas]
parts += [turn["content"] for turn in history]
parts += [doc["text"] for doc in retrieved_docs]
parts.append(user_message)
return sum(n_tokens(p) for p in parts)
record = {
"invoice_id": "INV-2026-00418",
"customer": {"id": "a3f9b2c1-77d4-4e8a-9b12-5c6d7e8f9a0b", "plan": "growth", "seats": 42},
"line_items": [
{"sku": "SKU-99183-A", "qty": 2, "unit_price_cents": 74950},
{"sku": "SKU-10277-C", "qty": 1, "unit_price_cents": 12900},
],
"subtotal_cents": 162800,
"tax_cents": 14652,
"total_cents": 177452,
"currency": "USD",
"due_date": "2026-04-14",
"status": "open",
}
as_json = json.dumps(record, separators=(",", ":"))
as_prose = (
"Invoice INV-2026-00418 for customer a3f9b2c1-77d4-4e8a-9b12-5c6d7e8f9a0b "
"on the growth plan with 42 seats is open, due 2026-04-14. Two units of "
"SKU-99183-A at 74950 cents and one unit of SKU-10277-C at 12900 cents. "
"Subtotal 162800, tax 14652, total 177452 USD."
)
print(f"json chars={len(as_json):>4} tokens={n_tokens(as_json):>4}")
print(f"prose chars={len(as_prose):>4} tokens={n_tokens(as_prose):>4}")
print(f"json costs {n_tokens(as_json) / n_tokens(as_prose):.2f}x the prose")
Expect that ratio to land between 1.4 and 2 times, and to drift as your key names get longer.
The second script projects a monthly bill, including a cache-hit assumption and a batch share.
from dataclasses import dataclass
RATES = {
"small": {"in": 0.25, "cached": 0.025, "out": 1.25},
"mid": {"in": 3.00, "cached": 0.30, "out": 15.00},
"frontier": {"in": 15.00, "cached": 1.50, "out": 75.00},
}
CACHE_WRITE_MULTIPLIER = 1.25 # a cache write usually costs slightly more than fresh input
BATCH_DISCOUNT = 0.50 # applied to uncached input and to output
CACHE_WRITES_PER_DAY = 288 # five-minute TTL, steady traffic
@dataclass
class Workload:
requests_per_day: int
cacheable_prefix_tokens: int # must be byte-identical AND first in the prompt
dynamic_input_tokens: int # history + retrieval + user message
output_tokens: int
cache_hit_ratio: float = 0.0
batch_share: float = 0.0
def monthly_cost(w: Workload, tier: str = "mid", days: int = 30) -> float:
r = RATES[tier]
reqs = w.requests_per_day * days
batch_reqs = reqs * w.batch_share
def slice_cost(n: float, discount: float) -> float:
cached = n * w.cacheable_prefix_tokens / 1e6 * w.cache_hit_ratio
uncached_prefix = n * w.cacheable_prefix_tokens / 1e6 * (1 - w.cache_hit_ratio)
dynamic = n * w.dynamic_input_tokens / 1e6
output = n * w.output_tokens / 1e6
return (
cached * r["cached"] # cache reads are already discounted
+ (uncached_prefix + dynamic) * r["in"] * discount
+ output * r["out"] * discount
)
writes = CACHE_WRITES_PER_DAY * days * w.cacheable_prefix_tokens / 1e6
write_cost = writes * r["in"] * CACHE_WRITE_MULTIPLIER * (w.cache_hit_ratio > 0)
return slice_cost(reqs - batch_reqs, 1.0) + slice_cost(batch_reqs, BATCH_DISCOUNT) + write_cost
base = dict(requests_per_day=20_000, cacheable_prefix_tokens=2_500,
dynamic_input_tokens=3_850, output_tokens=350)
scenarios = [
("baseline, no cache", Workload(**base)),
("cached prefix", Workload(**base, cache_hit_ratio=1.0)),
("cached + 30% batch", Workload(**base, cache_hit_ratio=1.0, batch_share=0.30)),
("cached, 0.9 hit rate", Workload(**base, cache_hit_ratio=0.90)),
]
for label, workload in scenarios:
print(f"{label:<22} {monthly_cost(workload):>10,.0f} USD / month")
That prints roughly 14,580, 10,611, 9,099 and 11,061 USD — the same four numbers derived by hand above. Change cache_hit_ratio first; it is the input most likely to be wrong.
Frequently asked questions
Do I pay for a request that fails?
Usually yes, at least partially: input tokens are billed once the prompt has been processed, and a provider returning a 500 after prefill has already done that work. Fail fast with a short timeout so you do not pay for a prefill that will be discarded, and do not retry requests that failed on validation rather than infrastructure.
Why did my bill double when my request count stayed flat?
Because tokens per request is not a constant. Usual causes, in rough order of frequency: history growing because nothing is evicted; retrieval returning more chunks after an index change; a tool catalogue growing from eight entries to thirty; a model upgrade that emits reasoning tokens where the previous version did not; and a template edit that broke the cache prefix. All five show up in a per-request token histogram and never in a request-count graph.
Is prompt caching the same as semantic caching?
No, and the confusion costs money. A prompt cache reuses key-value state for a byte-identical prefix inside the provider’s inference stack, and cannot return a wrong answer because it changes nothing about the computation. A semantic cache matches a new request against previously answered similar requests in your application, and can return a stale answer, so it needs a similarity threshold and an invalidation strategy. A semantic cache hit removes the request entirely; a prompt cache hit only discounts part of it.
Can I forecast a bill without running the workload?
Within a factor of two or three, which is not enough to price a product on. The uncertainty is not in the published rates but in tokens per request and requests per day, both of which depend on behaviour you have not observed yet. Build the instrumented version first and run it on real traffic for a week. If you need a number immediately, forecast the p95 rather than the mean.
Conclusion
The pricing model is three rates — fresh input, cached input, output — applied to counts you control. The arithmetic is not complicated, which is exactly why the surprises are embarrassing: they come from not counting something, not from mispricing it.
The single pricing decision that most often surprises teams is prompt layout, and it is made in week one by whoever writes the template. Putting a per-user instruction before a shared 2,000-token system prompt is reasonable for readability and it forfeits a discount of 80 to 90 percent on the largest token category in the request. Nothing breaks: the model still answers, and the invoice is simply larger than it needed to be for the life of the product.
So take the position explicitly. Order the prompt static-first and variable-last, treat the prefix as a versioned interface with a monitored hit ratio, cap reasoning effort per task class, route anything offline to the batch endpoint, and instrument real token counts from day one. Those five things turn a bill you discover into a number you predicted.
Leave a Reply