The bills that hurt are never the ones you planned for. A workload running at 200 USD a day does not become a 20,000 USD day because a model got more expensive; it becomes one because something looped, retried, or was invoked ten thousand times by a code path nobody was watching. Guardrails are a different problem from cost optimisation: optimisation lowers the baseline, while budget caps, kill switches and anomaly alerts bound the worst case.
Optimisation is not a control
Caching, model routing and prompt trimming reduce the price of a normal request. None of them stops an abnormal one. A 70 percent cache hit rate is a great number until an agent loop generates a unique prompt on every iteration and pays full price forever. Model routing does nothing when the runaway path legitimately needs the frontier model.
“Our cost per request is down 40 percent” and “we cannot spend more than 5,000 USD this month” are unrelated claims. The first is a cost reduction exercise; the second is admission control, and it must be enforced in the request path rather than observed afterwards in a dashboard.
Six failure modes that actually cause runaway bills
Runaway spend comes from a handful of structural bugs, each with an early signature and a cheap control.
An agent loop that never terminates. The model calls a tool, the tool returns something unusable, and the model calls the tool again. Nothing is broken in the traditional sense, so nothing alerts, and the cost curve is worse than linear because context grows each turn. Worked example: a 2,000-token system prompt plus 1,500 tokens per turn means input at turn n is 2,000 + 1,500(n-1). Over 200 turns that is 200 × 2,000 + 1,500 × (200 × 199 / 2) ≈ 30.25M input tokens, which at 3 USD per million is 90.75 USD for one conversation. A job fanning out 5,000 of them is 453,750 USD. The missing control is a maximum-turn and per-run token budget.
A retry storm. A retry wrapper with a short timeout, a reset backoff counter, or missing jitter can multiply one logical request into hundreds. Fifty workers retrying ten times a second against a failing call is 500 requests per second; at 0.02 USD each, roughly 36,000 USD per hour. Errors are cheap only if they fail fast — a call that burns input tokens before returning a 500 costs full price every attempt. Idempotency keys and a retry budget fix it; see the retry and 429 guide.
A user-triggered bulk action with no per-user cap. Someone selects “all 400,000 records” in a UI that fans out one model call per record. The org budget is fine; the individual account is not.
A runaway eval or backfill job. A scheduled job re-scores the last 90 days of traffic against a new prompt. It never appears in latency dashboards and runs at 02:00. Backfills cause more surprising months than anything else: one-off code, written quickly, with no budget.
A webhook redelivery loop. Your handler returns a 500 after the model call succeeds, the provider redelivers, and your handler calls the model again. Every redelivery is paid inference, and the signature is a burst of identical idempotency keys.
A prompt that grows the context window every turn. The slow burn: each turn appends, nothing is evicted, and per-request cost creeps up a few percent per turn. No per-request limit ever trips.
| Failure mode | Early signal | Control that stops it |
|---|---|---|
| Non-terminating agent loop | Turns per run climbing; tokens per run rising superlinearly | Max-turn cap, per-run token budget, spend-rate alert |
| Retry storm | Retries per logical request; requests per second per client; 5xx ratio | Retry token bucket, exponential backoff with full jitter, idempotency keys |
| Uncapped bulk action | Requests per user per minute; spend per user | Per-user daily cap, job-size ceiling, async queue with confirmation |
| Runaway eval or backfill | Off-hours spend; spend grouped by feature tag | Dedicated budget per feature, dry-run sample, hard job ceiling |
| Webhook redelivery loop | Duplicate idempotency keys; requests from one source per minute | Idempotency keys with a dedupe window, per-source rate cap |
| Growing context window | Tokens per request z-score; input-to-output ratio | Context compaction, sliding window, per-request token ceiling |
Budget hierarchy: put each control at the level that owns the failure
A single org-level budget is a smoke detector in a warehouse. It tells you the building is on fire and nothing about which aisle. Each level should exist to catch a failure the level above cannot localise.
| Level | Typical control | Why it belongs here |
|---|---|---|
| Org | Monthly hard ceiling, global kill switch, billing alerts | Last line of defence; the number finance signs off on |
| Team | Monthly budget with a named owner | Accountability without blocking unrelated teams |
| Environment | Separate budgets; development and staging capped hard | Stops an experiment or load test eating production headroom |
| User | Daily cap plus per-minute rate limit | Catches one account’s loop or bulk action while everyone else works |
| Feature | Budget per feature tag, e.g. summarise_v3 | Catches a bad deploy of one code path before it becomes an org incident |
| Request | max_tokens ceiling, context limit, pre-flight reservation | Bounds the worst possible single call |
Per-user caps catch what org caps structurally cannot. When an org budget trips, every user is blocked, including the 99.9 percent who behaved correctly, and the response starts with “who did this” — an investigation that takes hours when requests are untagged. A per-user cap turns the same event into one account hitting a limit. Enforce at the narrowest scope that covers the blast radius, and keep the wider scope as a backstop.
Soft caps, hard caps, and the 70/90/100 ladder
A hard cap with no warning is bad engineering: the first signal a customer gets is total failure, at the worst possible time, with no chance to react. A soft cap with no enforcement is not a control, just a notification people learn to filter. You need a ladder that both warns and enforces, applied per scope.
- 70 percent — warn. Alert the owning team, not the on-call rotation. Nothing is blocked. Increase trace sampling for this scope and put the top spenders by feature in the alert body.
- 90 percent — degrade. Shed cost automatically: route eligible traffic to a cheaper model, disable optional enrichment, tighten
max_tokens, stop accepting batch work. Users still get answers; they get smaller ones. - 100 percent — block. Reject new requests for that scope with a machine-readable error carrying the scope, the reset time, and a link to request an increase. In-flight requests finish; killing them mid-stream wastes paid tokens.
- 120 percent — kill. If spend still passes 120 percent of a hard limit, something bypassed the reservation path. Trip the kill switch and page a human: reaching this tier is a bug, not a budget decision.
One knob per scope, one owner, one place to look when behaviour changes.
Pre-flight estimation, atomic reservation, and reconciliation
You cannot enforce a budget you only learn about after the call. Every request needs a cost estimate before dispatch, a reservation against the relevant scopes, and a settlement once real usage is known.
Estimating before you call
Use the model’s real tokeniser when the request is large enough to matter, and a heuristic when it is not. For most chat traffic a character-based estimate lands within about 15 percent, which is fine for reservation as long as you reconcile: roughly four characters per token for Latin text, one and a half to two for CJK, plus 20 to 30 tokens of chat framing. Then price it using the output cap, not a hopeful average: estimate = input_tokens × price_in + max_tokens × price_out.
Reserving against max_tokens deliberately over-reserves, which is the correct default: over-reservation costs headroom, under-reservation costs the guarantee. If callers set max_tokens to 4096 and use 300, tighten the callers rather than weakening the guard.
The concurrency race
The naive implementation reads the remaining budget, compares it to the estimate, and writes the new total. Two requests arriving in the same millisecond both read 1.00 USD of headroom, both estimate 0.80 USD, both pass, and both spend — a 60 percent overrun from two concurrent calls, unbounded at real concurrency. This lost-update race is why homegrown guards fail precisely when traffic spikes.
Two fixes exist. An atomic counter performs the check and the increment in one operation, via a Redis Lua script or a conditional SQL update. A reservation ledger writes a row per request in a reserved state and moves it to settled afterwards. The counter is faster; the ledger gives attribution and post-hoc reconciliation. Ship both if you can, with the ledger as source of truth.
Where to enforce: application, gateway, or both
Application-level checks are the only place with enough context to know which user, feature and logical operation a request belongs to, and the easiest layer to bypass. Every new service, notebook, cron job and script is a fresh chance to call the provider without the guard.
A gateway sits in front of every provider call by construction and cannot be bypassed by new code as long as the credentials live behind it. That is the operational reason never to distribute raw provider keys: issue gateway keys with per-key budgets. The gateway knows the model, the token counts and the account, so it enforces org and tenant ceilings, rate limits and kill switches without caller cooperation.
Its blind spot is intent: it does not know this call is the 40th turn of a loop, or that it belongs to a backfill that should never exceed 200 USD. So the application should enforce semantic caps and pass a pre-flight estimate, while the gateway enforces the hard ceiling, the rate limit and the kill switch, and rejects requests arriving without a valid estimate header. That is what a gateway such as Qora API is built for: one key across providers with per-key budgets and unified metering.
Kill switches that are fast, safe, and rehearsed
A kill switch is a manual, deliberate stop. Unlike a circuit breaker, which is automatic and trips on error rate, a kill switch trips on money. It needs three properties.
Scoped. Support at least global, per-tenant, per-model and per-feature scopes. Stopping one tenant is a routine action; stopping everything is a company-level decision. If the only switch you have is global, nobody will pull it, which means it does not exist. The switch should stop new requests and let in-flight ones finish — except streaming calls, whose cost is unbounded until they end.
Fast. Read the flag from a cache with a short TTL, in the request path — not from a config deploy, not from a database query. A switch that takes four minutes to propagate arrives after the spend. Target seconds from flip to effect, and measure it.
Testable in production without a real incident. This is the property everyone skips and the one that decides whether the switch works when you need it. Run it in shadow mode — flip it for one internal tenant or one percent of traffic — and confirm callers get the expected error, retries do not amplify the rejection, and the alert fires. Do it on a schedule, because an untested kill switch is a boolean never evaluated under load.
Anomaly detection that catches the burn before the invoice
A per-request threshold misses slow burns by construction, and a daily alert arrives after the money is gone. Evaluate on minute-level aggregates, and prefer rules you actually ship.
- Spend rate versus trailing baseline. Compare spend in the last ten minutes against the median of the same bucket over the previous seven days. Alert when the ratio exceeds five and the value clears a floor, so 0.02 USD to 0.10 USD does not page anyone at 4am.
- Tokens per request z-score. Track mean and standard deviation of input tokens per feature over a rolling window; alert at mean plus three standard deviations. This catches the growing context window, which no cost-per-request rule sees.
- Requests per user per minute. Humans do not make 400 requests a minute; loops and bulk actions do.
- Cache hit-rate collapse. A drop from 60 percent to 5 percent means a cache-key bug or a prompt change that invalidated every entry. Both cost money and are invisible in latency metrics.
- Model-mix shift. Alert when the share of requests on your most expensive model moves by more than a few points. A routing bug sending everything to the frontier model is silent and expensive.
Alert on the derivative, not the level: a scope that always spends 500 USD a day should not page anyone for spending 500 USD a day. Put the top three spenders by tag in the alert body, because “spend is up 6x, and 92 percent of it is tenant_4471” ends an investigation that “spend is up 6x” starts. These are the aggregates described in the observability stack.
Graceful degradation beats hard failure
Blocking is the last resort. Most budget pressure can be absorbed by changing what the system does rather than whether it responds.
| Degradation | User impact | When to use |
|---|---|---|
| Fall back to a cheaper model | Lower quality on hard inputs, still correct on easy ones | Classification, extraction, summarisation, and any task where a cheap model passes your eval set |
| Disable an optional step | Less rich answer; nothing promised is missing | Enrichment such as reranking, second-opinion passes, speculative tool calls |
| Shorten the context | Long-range detail may be lost; recent turns preserved | When context is the cost driver and recency is what users rely on |
| Queue for later | Latency moves from seconds to hours | Non-interactive work: evals, backfills, report generation, embedding refreshes |
| Return a partial answer with a notice | Incomplete but honest and immediate | When the user prefers something now over everything later |
| Reject with a clear error | Work blocked until the budget resets or is raised | Only when the alternative is unbudgeted spend |
The routing decision must be precomputed, not improvised at 90 percent budget. Which tasks have an acceptable cheap-model fallback is a quality question that belongs in an eval run, not an incident. The model routing guide covers how to establish that mapping.
Attribution: tag every request or debug forever
Without tags, a spend spike is an unbounded investigation: a number, a timestamp, and hours of grepping logs for the code path that changed. With tags it is one query. Every request should carry tenant, user, feature, environment, model, and a trace identifier propagated through retries and tool calls, so a storm of 400 provider calls collapses into one logical request.
The trace identifier is the piece teams most often omit and the one that makes retry storms legible: without it, a storm looks like a traffic increase. The feature tag is the other high-value dimension because it maps onto a deploy, and GROUP BY feature ORDER BY usd DESC answers “what shipped” in seconds. The metering model behind this is described in the usage metering post.
A concrete implementation: reserve, then reconcile
The guard below estimates cost from the prompt and the output cap, reserves atomically against a scope, and reconciles with actual usage when the call returns. Atomicity comes from a Redis Lua script, so the read-check-write sequence cannot interleave.
import uuid
from dataclasses import dataclass
from decimal import Decimal
import redis
# Check and increment happen in one atomic step. Without this, two concurrent
# requests both read the same headroom and both pass.
RESERVE_LUA = """
local key = KEYS[1]
local amount = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local ttl = tonumber(ARGV[3])
local current = tonumber(redis.call('GET', key) or '0')
if current + amount > limit then
return {'-1', tostring(limit - current)}
end
local next_total = current + amount
redis.call('SET', key, next_total, 'EX', ttl)
return {tostring(next_total), tostring(limit - next_total)}
"""
@dataclass(frozen=True)
class Reservation:
request_id: str
scope: str
period: str
amount_usd: Decimal
class BudgetExceeded(Exception):
"""Raised when a scope has no headroom left for the estimated cost."""
class BudgetGuard:
def __init__(self, client: redis.Redis, limits: dict):
self.client = client
self.limits = limits # {(scope, period): Decimal}
self.reserve_script = client.register_script(RESERVE_LUA)
@staticmethod
def estimate_usd(prompt: str, max_output_tokens: int,
price_in_per_m: Decimal, price_out_per_m: Decimal) -> Decimal:
# Heuristic tokeniser: ~4 chars/token for Latin, ~1.7 for CJK,
# plus 24 tokens of chat framing. Reconcile afterwards, always.
ascii_chars = sum(1 for ch in prompt if ord(ch) < 128)
wide_chars = len(prompt) - ascii_chars
input_tokens = ascii_chars // 4 + int(wide_chars / 1.7) + 24
return (input_tokens * price_in_per_m
+ max_output_tokens * price_out_per_m) / Decimal(1_000_000)
def reserve(self, scope: str, period: str, amount_usd: Decimal,
seconds_left: int) -> Reservation:
limit = self.limits[(scope, period)]
key = "budget:{}:{}".format(period, scope)
total, remaining = self.reserve_script(
keys=[key],
args=[str(amount_usd), str(limit), int(seconds_left)],
)
if float(total) == -1:
raise BudgetExceeded(
"{} / {} has {:.4f} USD headroom left".format(scope, period, float(remaining))
)
return Reservation(uuid.uuid4().hex, scope, period, amount_usd)
def reconcile(self, reservation: Reservation, actual_usd: Decimal) -> None:
# Refund unused headroom, or charge the overrun. Redis preserves the TTL.
key = "budget:{}:{}".format(reservation.period, reservation.scope)
delta = actual_usd - reservation.amount_usd
if delta != 0:
self.client.incrbyfloat(key, str(delta))
# Call site: reserve against every scope that applies, innermost first.
guard = BudgetGuard(redis.Redis(), limits={
("org", "month"): Decimal("5000"),
("user:u_8812", "day"): Decimal("25"),
})
estimate = guard.estimate_usd(prompt, 800, Decimal("3.00"), Decimal("15.00"))
reservation = guard.reserve("user:u_8812", "day", estimate, seconds_left=43200)
try:
response = call_model(prompt, max_output_tokens=800)
finally:
# On failure actual usage is zero, so the reservation is refunded in full.
guard.reconcile(reservation, Decimal(str(response.usage.cost_usd)))The ledger is the durable half: the counter in front of it is a cache, the ledger is what you reconcile against the provider invoice. Note the FOR UPDATE on the limit row, which serialises concurrent reservations for the same scope and makes the conditional insert safe under load.
CREATE TABLE budget_reservations (
request_id uuid PRIMARY KEY,
trace_id uuid NOT NULL,
tenant_id text NOT NULL,
feature text NOT NULL,
model text NOT NULL,
scope text NOT NULL,
scope_id text NOT NULL,
period text NOT NULL,
period_start timestamptz NOT NULL,
reserved_usd numeric(14,6) NOT NULL,
settled_usd numeric(14,6),
state text NOT NULL DEFAULT 'reserved'
CHECK (state IN ('reserved', 'settled', 'released')),
created_at timestamptz NOT NULL DEFAULT now(),
settled_at timestamptz
);
CREATE INDEX budget_reservations_scope_idx
ON budget_reservations (scope, scope_id, period, period_start);
CREATE INDEX budget_reservations_trace_idx ON budget_reservations (trace_id);
CREATE INDEX budget_reservations_tenant_idx ON budget_reservations (tenant_id, created_at DESC);
-- Reserve. The row lock serialises concurrent reservations for this scope,
-- so two requests cannot both observe the same headroom.
BEGIN;
SELECT limit_usd FROM budget_limits
WHERE scope = :scope AND scope_id = :scope_id AND period = :period
FOR UPDATE;
INSERT INTO budget_reservations (
request_id, trace_id, tenant_id, feature, model,
scope, scope_id, period, period_start, reserved_usd
)
SELECT :request_id, :trace_id, :tenant_id, :feature, :model,
:scope, :scope_id, :period, :period_start, :estimate_usd
FROM (
SELECT :limit_usd - COALESCE(sum(COALESCE(settled_usd, reserved_usd)), 0) AS headroom
FROM budget_reservations
WHERE scope = :scope AND scope_id = :scope_id
AND period = :period AND period_start = :period_start
AND state <> 'released'
) AS h
WHERE h.headroom >= :estimate_usd
RETURNING request_id;
COMMIT;
-- Zero rows returned means the budget was exhausted: reject with HTTP 402 or 429.
-- Settle with real usage from the provider response.
UPDATE budget_reservations
SET state = 'settled', settled_usd = :actual_usd, settled_at = now()
WHERE request_id = :request_id AND state = 'reserved';
-- Attribution: what is spending, right now, for this tenant.
SELECT feature, model,
count(*) AS calls,
sum(COALESCE(settled_usd, reserved_usd)) AS usd
FROM budget_reservations
WHERE tenant_id = :tenant_id
AND created_at >= now() - interval '15 minutes'
GROUP BY feature, model
ORDER BY usd DESC
LIMIT 10;The minimum viable control set for week one
With one week, I would ship five things and deliberately skip the rest.
- Tag every request with tenant, user, feature and trace id. Nothing else works without this.
- Reserve and reconcile against an atomic counter at the org and per-user scopes. Two scopes, one script, one call site.
- Set the 70/90/100 ladder with a real degradation path at 90 percent. If the 90 percent tier does nothing, you have not shipped a ladder.
- Ship one global and one per-tenant kill switch, read from a cached flag, with a documented flip procedure and a shadow-mode test you have actually run.
- Add three alerts: spend rate versus trailing baseline, tokens per request z-score, and requests per user per minute — minute-level evaluation, top spenders in the alert body.
Defer per-feature budgets until the taxonomy stabilises, the 120 percent tier until you trust reservations, and ML-based anomaly detection entirely: rules catch the failure modes above, and an untrusted alert is an ignored alert.
Frequently asked questions
Should the budget be enforced in the application or at the gateway?
Both, with different jobs. The application owns the semantic caps only it can see — per-user daily limits, per-feature budgets, maximum turns per agent run — and passes a pre-flight estimate downstream. The gateway owns the ceilings that must hold regardless of which code path is calling. Application-only enforcement fails the moment someone adds a service that calls the provider directly, and it fails silently. If you ship one first, ship the gateway: it is the only layer that cannot be bypassed.
How accurate does pre-flight estimation need to be?
Accurate enough that the reservation is a useful bound. A character-based heuristic lands within roughly 15 percent for typical chat traffic, and reservation over-reserves anyway because it charges max_tokens rather than expected output. Reconciliation keeps the system honest: reserved amounts are provisional, settled amounts are truth, and the counter is corrected by the delta. Estimates consistently off by more than 30 percent point at your callers’ max_tokens settings, not at the guard.
Why not just set a hard cap and be done with it?
Because the first thing users experience is total failure with no warning, usually at an inconvenient hour. A hard cap with no ladder also fails badly at the org level: one runaway job exhausts the shared budget and every other tenant is blocked, turning a single bad deploy into a platform-wide outage. The ladder exists so the system changes behaviour before it stops responding.
How do I know my kill switch will actually work?
By using it in production when nothing is wrong. Flip it for an internal tenant or one percent of traffic on a schedule, verify callers receive the expected error code and that retries do not amplify the rejection into a storm, then time the interval from flip to effect and treat it as an SLO. A kill switch never exercised under production load is an untested branch in the most important code path you own.
Conclusion
Runaway AI spend is a bounded problem with a small set of known causes. The failure modes are structural — loops without termination, retries without budgets, bulk actions without per-user caps, jobs without ceilings — and each has a signature that appears in aggregates long before it appears on an invoice. What makes guardrails work is coverage, not sophistication: tag everything, reserve atomically before dispatching, reconcile after, enforce at the narrowest scope that covers the blast radius, and keep a gateway as the line that cannot be bypassed.
Leave a Reply