{"id":102,"date":"2026-09-16T23:58:14","date_gmt":"2026-09-16T15:58:14","guid":{"rendered":"https:\/\/wp.qoraapi.com\/llm-observability\/"},"modified":"2026-09-20T03:53:19","modified_gmt":"2026-09-19T19:53:19","slug":"llm-observability","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/llm-observability\/","title":{"rendered":"LLM Observability: Monitoring AI API Usage, Latency and Cost"},"content":{"rendered":"\n

LLM observability<\/strong> is the practice of capturing a structured record of every AI API request \u2014 model, prompt and completion tokens, latency, cost, and error type \u2014 so you can see what your models actually cost, how fast they respond, and where they fail. It combines three things: request tracing, time-series metrics, and token-level cost accounting.<\/p>\n\n\n\n

Without it, every question about your AI spend is a guess. With it, questions like “which feature is burning the budget” and “did last night’s prompt change make responses slower” become queries you run in seconds.<\/p>\n\n\n\n

Why AI APIs are harder to observe than normal endpoints<\/h2>\n\n\n\n

A conventional web request is easy to monitor: status code, duration, payload size. An LLM call breaks all three assumptions. It returns HTTP 200 even when the answer is wrong. Its duration depends on how many tokens it chose to generate, not on how much work your server did. And its cost is variable \u2014 the same endpoint can bill you a fraction of a cent or several cents depending on what the model decided to write.<\/p>\n\n\n\n

That last point is the one that catches teams out. With a normal API, a traffic spike and a cost spike are the same event. With an LLM API, cost is driven by token volume<\/em>, which is driven by prompt design, retrieved context size, and how verbose the model decides to be. A feature that is barely used can dominate the bill. Observability is how you find out before the invoice does.<\/p>\n\n\n\n

The five signals worth capturing<\/h2>\n\n\n\n

You can log everything, but only a handful of fields change decisions. Capture these on every call:<\/p>\n\n\n\n

Signal<\/th>What it tells you<\/th>Typical use<\/th><\/tr><\/thead>
Model + provider<\/td>Which endpoint actually answered<\/td>Routing audits, A\/B comparisons<\/td><\/tr>
Input \/ output tokens<\/td>Volume, and therefore cost<\/td>Budgets, per-feature attribution<\/td><\/tr>
Time to first token (TTFT)<\/td>Perceived responsiveness<\/td>UX tuning, streaming health<\/td><\/tr>
Total latency<\/td>Wall-clock duration<\/td>Timeouts, retry policy tuning<\/td><\/tr>
Status \/ error class<\/td>429, 5xx, timeout, content filter<\/td>Reliability and capacity planning<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Two more fields are cheap to add and disproportionately useful: a request ID<\/strong> so you can trace one user action across multiple model calls, and a feature or route tag<\/strong> so you can attribute spend to the part of the product that caused it. Without the tag, all your cost data collapses into one undifferentiated number.<\/p>\n\n\n\n

Instrumenting a call<\/h2>\n\n\n\n

The good news is that OpenAI-compatible APIs return token usage in the response, so instrumentation is a wrapper rather than a rewrite. Here is the minimal version that captures everything in the table above:<\/p>\n\n\n\n

import time, logging, json\n\nlog = logging.getLogger(\"llm\")\n\ndef observed_call(client, *, feature, model, messages, **kw):\n    t0 = time.perf_counter()\n    ttft = None\n    try:\n        resp = client.chat.completions.create(model=model, messages=messages, **kw)\n        ttft = time.perf_counter() - t0          # non-streaming: TTFT ~= total\n\n        u = resp.usage\n        log.info(json.dumps({\n            \"event\":      \"llm_call\",\n            \"feature\":    feature,               # cost attribution tag\n            \"model\":      resp.model,            # what actually answered\n            \"in_tokens\":  u.prompt_tokens,\n            \"out_tokens\": u.completion_tokens,\n            \"ttft_ms\":    round(ttft * 1000, 1),\n            \"total_ms\":   round((time.perf_counter() - t0) * 1000, 1),\n            \"status\":     \"ok\",\n        }))\n        return resp\n\n    except Exception as e:\n        log.warning(json.dumps({\n            \"event\":    \"llm_call\",\n            \"feature\":  feature,\n            \"model\":    model,\n            \"total_ms\": round((time.perf_counter() - t0) * 1000, 1),\n            \"status\":   type(e).__name__,        # RateLimitError, APITimeoutError, ...\n        }))\n        raise\n<\/code><\/pre>\n\n\n\n
Two details make this worth more than a naive log line. First, the model is read from the response<\/em>, not from your request \u2014 a relay or gateway may route to a different model than you asked for, and you want your cost data to reflect reality. Second, failures are logged with the same shape as successes, so error rates and latency percentiles come from one dataset instead of two.<\/p>\n\n\n\n
Latency: measure TTFT, not just the average<\/h2>\n\n\n\n
Average latency is the most misleading metric in AI products. A chatbot that streams its first token in 400 ms and finishes in 6 seconds has an average of over 3 seconds, yet users perceive it as fast. A batch summarizer that takes 9 seconds and shows nothing until it is done has the same average and feels broken.<\/p>\n\n\n\n
Split the metric in two. TTFT<\/strong> governs perceived speed and should be tracked at p50, p95, and p99. Total duration<\/strong> governs throughput, timeouts, and retry behavior. Track them separately and you will stop “optimizing” latency in ways that make the product feel worse.<\/p>\n\n\n\n
Streaming also changes what you can measure. When the response arrives as Server-Sent Events, the final usage block may only be present if you explicitly request it, so instrument the stream itself rather than waiting for a single response object. Our streaming and SSE guide<\/a> covers the wire format and where the usage counts appear.<\/p>\n\n\n\n
Token tracking and cost dashboards<\/h2>\n\n\n\n
Cost is the metric that gets executive attention, so build it properly. Never hard-code prices into your dashboard; store tokens and multiply by a price table you update in one place. Prices change, and a dashboard that silently reports stale numbers is worse than no dashboard.<\/p>\n\n\n\n
Dashboard panel<\/th>Breakdown<\/th>Question it answers<\/th><\/tr><\/thead>
Spend over time<\/td>Daily, by feature<\/td>Are we trending toward a runaway?<\/td><\/tr>
Cost per request<\/td>p50 \/ p95 by model<\/td>Is a prompt change inflating prompts?<\/td><\/tr>
Token mix<\/td>Input vs output tokens<\/td>Are we paying for context we do not need?<\/td><\/tr>
Latency<\/td>TTFT and total, p95<\/td>Did the last deploy make us slower?<\/td><\/tr>
Error rate<\/td>By class (429, 5xx, timeout)<\/td>Are we capacity-limited or buggy?<\/td><\/tr>
Top consumers<\/td>By user or API key<\/td>Who is generating the volume?<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
If you build only two of these panels, build spend over time by feature<\/em> and token mix<\/em>. The first finds the runaway. The second explains it: input tokens climbing while output stays flat almost always means retrieved context is growing \u2014 the classic RAG leak. Our AI API cost reduction guide<\/a> covers the fixes, from prompt compression to caching and routing.<\/p>\n\n\n\n
Error telemetry: 429s are a capacity signal<\/h2>\n\n\n\n
Error class matters more than error count. A rising 429 rate is not a bug in your code \u2014 it is your provider telling you that you are exceeding a rate or quota limit, and it usually precedes a user-visible outage. A rising timeout rate often means a specific model has degraded or your own retry logic is amplifying load. A rising content-filter rate is a product signal about what your users are sending.<\/p>\n\n\n\n
Tag each class separately, and alert on the rate of change rather than the absolute number. The handling patterns \u2014 exponential backoff with jitter, fallback chains, and request queues \u2014 are covered in our guide to AI API rate limits and 429 errors<\/a>.<\/p>\n\n\n\n
Tracing multi-step and RAG requests<\/h2>\n\n\n
Single calls are easy. The interesting failures happen in chains: retrieve documents, re-rank, summarize, then answer. When that pipeline gets slow or expensive, an aggregate metric cannot tell you which stage is responsible.<\/p>\n\n\n\n
The fix is a trace: one trace ID per user action, one span per model call, with parent-child relationships. A trace view turns “the assistant got slow” into “the re-ranking call grew from 4 retrieved chunks to 40.” You do not need a heavyweight platform to start \u2014 a correlation ID in your logs, plus a table of spans, gets you 80% of the value on day one.<\/p>\n\n\n\n
Prompts, privacy, and retention<\/h2>\n\n\n\n
There is a version of this that goes wrong: a team logs full prompts and completions to make debugging easier, and three months later discovers customer data sitting in a log index with no retention policy. Observability and data protection have to be designed together.<\/p>\n\n\n\n
A workable split is to always store metadata<\/em> and to sample content<\/em>. Metadata \u2014 tokens, latency, model, status, feature tag \u2014 carries almost all the operational value and contains no user data. Prompt and completion text is what you need for a narrow class of debugging, so store it for a short window, on a sample, or behind an explicit flag that is off by default. Hash or drop anything that looks like an identifier before it leaves your process.<\/p>\n\n\n\n
Retention follows from the same logic. Cost and latency aggregates should live for a year or more, because they are how you spot slow drift. Raw payloads should expire in days. Teams that separate the two find they can answer nearly every operational question without ever keeping a customer’s conversation on disk.<\/p>\n\n\n\n
A minimal event schema you can standardise on<\/h2>\n\n\n\n
Whichever backend you use, agree on one event shape. It makes dashboards reusable, makes switching providers a config change, and stops the slow drift into five incompatible log formats:<\/p>\n\n\n\n
{\n  \"event\":       \"llm_call\",\n  \"trace_id\":    \"a41f9c2e\",          \/\/ one per user action\n  \"span_id\":     \"span-3\",            \/\/ one per model call in the chain\n  \"feature\":     \"chat.support\",      \/\/ cost attribution tag\n  \"model\":       \"gpt-4o-mini\",       \/\/ from the response, not the request\n  \"provider\":    \"relay\",             \/\/ who actually served it\n  \"in_tokens\":   1840,\n  \"out_tokens\":  212,\n  \"ttft_ms\":     412,\n  \"total_ms\":    2317,\n  \"status\":      \"ok\",                \/\/ or RateLimitError, APITimeoutError\n  \"streamed\":    true,\n  \"ts\":          \"2026-09-16T15:55:02Z\"\n}\n<\/code><\/pre>\n\n\n\n
Six of those fields drive every chart in this article. The other six exist so that when something goes wrong at 3 a.m. you can go from an alert to the exact request without guesswork. Build the schema once, keep it boring, and resist the urge to add fields that only one dashboard reads.<\/p>\n\n\n\n
Alerts that actually catch runaways<\/h2>\n\n\n\n