usage metering post<\/a>.<\/p>\n\n\n\nA concrete implementation: reserve, then reconcile<\/h2>\n\n\n\n
The guard below estimates cost from the prompt and the output cap, reserves atomically against a scope, and reconciles with actual usage when the call returns. Atomicity comes from a Redis Lua script, so the read-check-write sequence cannot interleave.<\/p>\n\n\n\n
import uuid\nfrom dataclasses import dataclass\nfrom decimal import Decimal\n\nimport redis\n\n# Check and increment happen in one atomic step. Without this, two concurrent\n# requests both read the same headroom and both pass.\nRESERVE_LUA = \"\"\"\nlocal key = KEYS[1]\nlocal amount = tonumber(ARGV[1])\nlocal limit = tonumber(ARGV[2])\nlocal ttl = tonumber(ARGV[3])\nlocal current = tonumber(redis.call('GET', key) or '0')\nif current + amount > limit then\n return {'-1', tostring(limit - current)}\nend\nlocal next_total = current + amount\nredis.call('SET', key, next_total, 'EX', ttl)\nreturn {tostring(next_total), tostring(limit - next_total)}\n\"\"\"\n\n@dataclass(frozen=True)\nclass Reservation:\n request_id: str\n scope: str\n period: str\n amount_usd: Decimal\n\nclass BudgetExceeded(Exception):\n \"\"\"Raised when a scope has no headroom left for the estimated cost.\"\"\"\n\nclass BudgetGuard:\n def __init__(self, client: redis.Redis, limits: dict):\n self.client = client\n self.limits = limits # {(scope, period): Decimal}\n self.reserve_script = client.register_script(RESERVE_LUA)\n\n @staticmethod\n def estimate_usd(prompt: str, max_output_tokens: int,\n price_in_per_m: Decimal, price_out_per_m: Decimal) -> Decimal:\n # Heuristic tokeniser: ~4 chars\/token for Latin, ~1.7 for CJK,\n # plus 24 tokens of chat framing. Reconcile afterwards, always.\n ascii_chars = sum(1 for ch in prompt if ord(ch) < 128)\n wide_chars = len(prompt) - ascii_chars\n input_tokens = ascii_chars \/\/ 4 + int(wide_chars \/ 1.7) + 24\n return (input_tokens * price_in_per_m\n + max_output_tokens * price_out_per_m) \/ Decimal(1_000_000)\n\n def reserve(self, scope: str, period: str, amount_usd: Decimal,\n seconds_left: int) -> Reservation:\n limit = self.limits[(scope, period)]\n key = \"budget:{}:{}\".format(period, scope)\n total, remaining = self.reserve_script(\n keys=[key],\n args=[str(amount_usd), str(limit), int(seconds_left)],\n )\n if float(total) == -1:\n raise BudgetExceeded(\n \"{} \/ {} has {:.4f} USD headroom left\".format(scope, period, float(remaining))\n )\n return Reservation(uuid.uuid4().hex, scope, period, amount_usd)\n\n def reconcile(self, reservation: Reservation, actual_usd: Decimal) -> None:\n # Refund unused headroom, or charge the overrun. Redis preserves the TTL.\n key = \"budget:{}:{}\".format(reservation.period, reservation.scope)\n delta = actual_usd - reservation.amount_usd\n if delta != 0:\n self.client.incrbyfloat(key, str(delta))\n\n# Call site: reserve against every scope that applies, innermost first.\nguard = BudgetGuard(redis.Redis(), limits={\n (\"org\", \"month\"): Decimal(\"5000\"),\n (\"user:u_8812\", \"day\"): Decimal(\"25\"),\n})\n\nestimate = guard.estimate_usd(prompt, 800, Decimal(\"3.00\"), Decimal(\"15.00\"))\nreservation = guard.reserve(\"user:u_8812\", \"day\", estimate, seconds_left=43200)\ntry:\n response = call_model(prompt, max_output_tokens=800)\nfinally:\n # On failure actual usage is zero, so the reservation is refunded in full.\n guard.reconcile(reservation, Decimal(str(response.usage.cost_usd)))<\/code><\/pre>\n\n\n\nThe ledger is the durable half: the counter in front of it is a cache, the ledger is what you reconcile against the provider invoice. Note the FOR UPDATE<\/code> on the limit row, which serialises concurrent reservations for the same scope and makes the conditional insert safe under load.<\/p>\n\n\n\nCREATE TABLE budget_reservations (\n request_id uuid PRIMARY KEY,\n trace_id uuid NOT NULL,\n tenant_id text NOT NULL,\n feature text NOT NULL,\n model text NOT NULL,\n scope text NOT NULL,\n scope_id text NOT NULL,\n period text NOT NULL,\n period_start timestamptz NOT NULL,\n reserved_usd numeric(14,6) NOT NULL,\n settled_usd numeric(14,6),\n state text NOT NULL DEFAULT 'reserved'\n CHECK (state IN ('reserved', 'settled', 'released')),\n created_at timestamptz NOT NULL DEFAULT now(),\n settled_at timestamptz\n);\n\nCREATE INDEX budget_reservations_scope_idx\n ON budget_reservations (scope, scope_id, period, period_start);\nCREATE INDEX budget_reservations_trace_idx ON budget_reservations (trace_id);\nCREATE INDEX budget_reservations_tenant_idx ON budget_reservations (tenant_id, created_at DESC);\n\n-- Reserve. The row lock serialises concurrent reservations for this scope,\n-- so two requests cannot both observe the same headroom.\nBEGIN;\n\nSELECT limit_usd FROM budget_limits\nWHERE scope = :scope AND scope_id = :scope_id AND period = :period\nFOR UPDATE;\n\nINSERT INTO budget_reservations (\n request_id, trace_id, tenant_id, feature, model,\n scope, scope_id, period, period_start, reserved_usd\n)\nSELECT :request_id, :trace_id, :tenant_id, :feature, :model,\n :scope, :scope_id, :period, :period_start, :estimate_usd\nFROM (\n SELECT :limit_usd - COALESCE(sum(COALESCE(settled_usd, reserved_usd)), 0) AS headroom\n FROM budget_reservations\n WHERE scope = :scope AND scope_id = :scope_id\n AND period = :period AND period_start = :period_start\n AND state <> 'released'\n) AS h\nWHERE h.headroom >= :estimate_usd\nRETURNING request_id;\n\nCOMMIT;\n-- Zero rows returned means the budget was exhausted: reject with HTTP 402 or 429.\n\n-- Settle with real usage from the provider response.\nUPDATE budget_reservations\nSET state = 'settled', settled_usd = :actual_usd, settled_at = now()\nWHERE request_id = :request_id AND state = 'reserved';\n\n-- Attribution: what is spending, right now, for this tenant.\nSELECT feature, model,\n count(*) AS calls,\n sum(COALESCE(settled_usd, reserved_usd)) AS usd\nFROM budget_reservations\nWHERE tenant_id = :tenant_id\n AND created_at >= now() - interval '15 minutes'\nGROUP BY feature, model\nORDER BY usd DESC\nLIMIT 10;<\/code><\/pre>\n\n\n\nThe minimum viable control set for week one<\/h2>\n\n\n\n
With one week, I would ship five things and deliberately skip the rest.<\/p>\n\n\n\n
- Tag every request<\/strong> with tenant, user, feature and trace id. Nothing else works without this.<\/li>
- Reserve and reconcile against an atomic counter<\/strong> at the org and per-user scopes. Two scopes, one script, one call site.<\/li>
- Set the 70\/90\/100 ladder<\/strong> with a real degradation path at 90 percent. If the 90 percent tier does nothing, you have not shipped a ladder.<\/li>
- Ship one global and one per-tenant kill switch<\/strong>, read from a cached flag, with a documented flip procedure and a shadow-mode test you have actually run.<\/li>
- Add three alerts<\/strong>: spend rate versus trailing baseline, tokens per request z-score, and requests per user per minute \u2014 minute-level evaluation, top spenders in the alert body.<\/li><\/ol>\n\n\n\n
Defer per-feature budgets until the taxonomy stabilises, the 120 percent tier until you trust reservations, and ML-based anomaly detection entirely: rules catch the failure modes above, and an untrusted alert is an ignored alert.<\/p>\n\n\n\n
Frequently asked questions<\/h2>\n\n\n\nShould the budget be enforced in the application or at the gateway?<\/h3>\n\n\n\n
Both, with different jobs. The application owns the semantic caps only it can see \u2014 per-user daily limits, per-feature budgets, maximum turns per agent run \u2014 and passes a pre-flight estimate downstream. The gateway owns the ceilings that must hold regardless of which code path is calling. Application-only enforcement fails the moment someone adds a service that calls the provider directly, and it fails silently. If you ship one first, ship the gateway: it is the only layer that cannot be bypassed.<\/p>\n\n\n\n
How accurate does pre-flight estimation need to be?<\/h3>\n\n\n\n
Accurate enough that the reservation is a useful bound. A character-based heuristic lands within roughly 15 percent for typical chat traffic, and reservation over-reserves anyway because it charges max_tokens<\/code> rather than expected output. Reconciliation keeps the system honest: reserved amounts are provisional, settled amounts are truth, and the counter is corrected by the delta. Estimates consistently off by more than 30 percent point at your callers’ max_tokens<\/code> settings, not at the guard.<\/p>\n\n\n\nWhy not just set a hard cap and be done with it?<\/h3>\n\n\n\n
Because the first thing users experience is total failure with no warning, usually at an inconvenient hour. A hard cap with no ladder also fails badly at the org level: one runaway job exhausts the shared budget and every other tenant is blocked, turning a single bad deploy into a platform-wide outage. The ladder exists so the system changes behaviour before it stops responding.<\/p>\n\n\n\n
How do I know my kill switch will actually work?<\/h3>\n\n\n\n
By using it in production when nothing is wrong. Flip it for an internal tenant or one percent of traffic on a schedule, verify callers receive the expected error code and that retries do not amplify the rejection into a storm, then time the interval from flip to effect and treat it as an SLO. A kill switch never exercised under production load is an untested branch in the most important code path you own.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
Runaway AI spend is a bounded problem with a small set of known causes. The failure modes are structural \u2014 loops without termination, retries without budgets, bulk actions without per-user caps, jobs without ceilings \u2014 and each has a signature that appears in aggregates long before it appears on an invoice. What makes guardrails work is coverage, not sophistication: tag everything, reserve atomically before dispatching, reconcile after, enforce at the narrowest scope that covers the blast radius, and keep a gateway as the line that cannot be bypassed.<\/p>\n\n\n\n\n