{"id":166,"date":"2026-09-20T02:37:09","date_gmt":"2026-09-19T18:37:09","guid":{"rendered":"https:\/\/wp.qoraapi.com\/load-testing-llm-apps\/"},"modified":"2026-09-20T02:49:52","modified_gmt":"2026-09-19T18:49:52","slug":"load-testing-llm-apps","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/load-testing-llm-apps\/","title":{"rendered":"Load Testing LLM Apps: Throughput, TTFT, and Concurrency"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Load testing an LLM app means ramping concurrency in steps while recording time to first token (TTFT), inter-token latency, throughput in tokens per second, and error rate \u2014 then finding the concurrency at which p95 TTFT stops being flat. REST-style RPS testing misses this entirely, because LLM latency and cost scale with generated tokens, not with requests.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why LLM load testing is different from REST load testing<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A REST handler&#8217;s service time is roughly constant and independent of payload, so you size capacity in requests per second and latency stays flat until a resource saturates. LLM endpoints break that model in four ways.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Output length is a random variable.<\/strong> A REST handler returns a fixed-size row; an LLM returns however many tokens it decides to emit. Since end-to-end latency \u2248 TTFT + tokens \u00d7 per-token time, latency and cost are both random variables. One fixed prompt samples a distribution and says nothing about p95.<\/li>\n<li><strong>Streaming holds the connection open for the whole generation.<\/strong> A REST call occupies a worker for milliseconds; a streaming generation holds an in-flight slot for 4\u201320 seconds. Concurrency here means concurrent generations, governed by Little&#8217;s Law: in-flight = arrival rate \u00d7 mean service time.<\/li>\n<li><strong>The provider queues on your behalf.<\/strong> Your process can sit at 5% CPU while p95 TTFT triples, because the queue is on someone else&#8217;s infrastructure. Local resource metrics are useless as a saturation signal; the only honest instrument is client-side timing.<\/li>\n<li><strong>Load tests have an invoice.<\/strong> Cost scales with tokens generated, so a test emitting 10\u00d7 more output tokens costs 10\u00d7 more \u2014 well designed or not.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Provider limits are usually enforced per key on both requests per minute and tokens per minute, so testing on the credential that serves live users trips the ceiling for real traffic \u2014 use a separate key, and see our guide to <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-rate-limits-429-errors\/\">rate limits<\/a>. The upshot: capacity extrapolated from a single-prompt, non-streaming, low-concurrency test is wrong in the optimistic direction \u2014 the dangerous direction.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The metrics that actually matter<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Six metrics carry almost all the information. Measure them per request on the client, then aggregate per concurrency step \u2014 never as one mean.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Metric<\/th><th>Precise definition<\/th><th>How to measure it<\/th><th>What it diagnoses<\/th><\/tr><\/thead><tbody><tr><td>TTFT (time to first token)<\/td><td>Send to first chunk with non-empty content<\/td><td>Timestamp before the HTTP call; timestamp the first SSE delta with content<\/td><td>Provider queueing + prefill<\/td><\/tr><tr><td>TPOT \/ inter-token latency<\/td><td>(end-to-end \u2212 TTFT) \u00f7 (output_tokens \u2212 1)<\/td><td>Derive per request from the two timestamps and the token count<\/td><td>Decode speed; rises when the provider batches harder<\/td><\/tr><tr><td>End-to-end latency<\/td><td>Send to final token<\/td><td>Client-side timer around the whole stream<\/td><td>Batch-job and non-streaming UX<\/td><\/tr><tr><td>Throughput<\/td><td>Output tokens \u00f7 wall-clock seconds of the step<\/td><td>Aggregate over the hold window<\/td><td>Capacity. Better than RPS, which is not portable across prompt mixes<\/td><\/tr><tr><td>Goodput<\/td><td>Requests meeting <em>both<\/em> the TTFT and end-to-end SLOs<\/td><td>Count per step against your SLO thresholds<\/td><td>Usable capacity \u2014 high throughput with blown TTFT is not shippable<\/td><\/tr><tr><td>Cost per request<\/td><td>(input_tokens \u00d7 input_ratio) + (output_tokens \u00d7 output_ratio)<\/td><td>Token counters from the API, weighted by relative tier pricing<\/td><td>Test-budget predictability; output-length drift<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Two rules make these numbers trustworthy. First, <strong>report percentiles, never means<\/strong>: LLM latency distributions are heavy-tailed, and a few requests stuck behind a provider queue drag a mean around. Report p50, p95, and p99.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Second, <strong>never collapse TTFT into end-to-end.<\/strong> TTFT is dominated by queue wait plus prefill of your input; the remainder by decode. Halving your system prompt improves TTFT and leaves TPOT untouched; a provider raising its batch size does the reverse.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Designing a realistic load test<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>1. Sample the prompt mix from production, weighted by traffic share.<\/strong> Bucket real inputs by input-token count \u2014 short under 200, medium 200\u20131500, long over 1500 \u2014 and weight each bucket by its traffic share, using at least 50 distinct prompts per bucket or a random nonce. Replaying one identical prompt thousands of times triggers prompt caching, so you measure an artificially fast, artificially cheap system that does not exist for real users.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>2. Cap max_tokens at your production cap.<\/strong> Uncapped outputs turn a 60-second step into a multi-thousand-token step and blow the budget. The cap is what production uses, so it belongs in the test.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>3. Model think time, and know whether you are closed-loop or open-loop.<\/strong> A closed-loop harness with a fixed worker count self-throttles: as latency rises, each worker completes fewer requests, offered load silently drops, and you under-report queueing. Finding the true knee needs an open-loop run at a fixed arrival rate, where offered load stays constant while latency grows.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>4. Ramp in steps and discard warmup.<\/strong> Use a geometric ramp (1, 2, 4, 8, 16, 32, 64), hold each step 60\u2013120 seconds, and discard the first 15\u201330 seconds. Shorter holds measure connection pool filling, not steady state.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This harness ramps concurrency, streams every request, records TTFT per request, and prints a percentile summary per step. Install with <code>pip install httpx<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import asyncio, json, random, time\nimport httpx\n\nURL   = \"https:\/\/your-gateway\/v1\/chat\/completions\"\nKEY   = \"sk-...\"                 # a dedicated load-test key, NOT the production one\nMODEL = \"cheap-small-tier-model\" # ramp on the cheap tier, confirm on the real one\n\n# (weight, prompt, max_tokens) sampled from the production length distribution\nMIX = [\n    (0.60, \"Classify the sentiment of this review: ...\", 32),\n    (0.30, \"Summarize this support ticket in three bullets: ...\", 200),\n    (0.10, \"Extract every line item into JSON: ...\", 700),\n]\nRAMP, HOLD_S, WARMUP_S, THINK_S = [1, 2, 4, 8, 16, 32, 64], 60, 15, 2.0\nBUDGET_TOKENS = 400_000          # hard stop so the test cannot run away\nspent = 0\n\ndef sample(rng):\n    r, acc = rng.random(), 0.0\n    for w, text, mt in MIX:\n        acc += w\n        if r &lt;= acc:\n            return text, mt\n    return MIX[-1][1], MIX[-1][2]\n\nasync def one(client, rng, rec):\n    global spent\n    text, max_tokens = sample(rng)\n    body = {\"model\": MODEL, \"stream\": True, \"max_tokens\": max_tokens,\n            \"messages\": [{\"role\": \"user\", \"content\": text}]}\n    t0 = time.perf_counter()\n    ttft = None\n    toks = 0\n    try:\n        async with client.stream(\"POST\", URL, json=body,\n                                 headers={\"Authorization\": f\"Bearer {KEY}\"}) as r:\n            r.raise_for_status()\n            async for line in r.aiter_lines():\n                if not line.startswith(\"data:\"):\n                    continue\n                chunk = line[5:].strip()\n                if chunk == \"[DONE]\":\n                    break\n                delta = json.loads(chunk)[\"choices\"][0].get(\"delta\", {})\n                if delta.get(\"content\"):\n                    toks += 1\n                    if ttft is None:\n                        ttft = time.perf_counter() - t0   # first CONTENT token\n        rec.append({\"ok\": True, \"ttft\": ttft, \"e2e\": time.perf_counter() - t0,\n                    \"tokens\": toks, \"t\": time.perf_counter()})\n        spent += toks\n    except Exception as e:\n        rec.append({\"ok\": False, \"err\": type(e).__name__, \"t\": time.perf_counter()})\n\nasync def worker(client, rng, rec, stop):\n    while not stop.is_set():\n        await one(client, rng, rec)\n        await asyncio.sleep(rng.expovariate(1 \/ THINK_S))   # user think time\n\nasync def step(concurrency):\n    rec, stop, rng = [], asyncio.Event(), random.Random(42)\n    limits = httpx.Limits(max_connections=concurrency,\n                          max_keepalive_connections=concurrency)\n    async with httpx.AsyncClient(timeout=120, limits=limits) as client:\n        tasks = [asyncio.create_task(worker(client, rng, rec, stop))\n                 for _ in range(concurrency)]\n        await asyncio.sleep(WARMUP_S)\n        cut = len(rec)                       # discard warmup samples\n        await asyncio.sleep(HOLD_S)\n        stop.set()\n        await asyncio.gather(*tasks, return_exceptions=True)\n    return rec[cut:]\n\ndef pct(xs, p):\n    xs = sorted(xs)\n    return xs[min(len(xs) - 1, int(len(xs) * p))] if xs else float(\"nan\")\n\nasync def main():\n    for c in RAMP:\n        if spent &gt; BUDGET_TOKENS:\n            print(json.dumps({\"aborted\": \"token budget exhausted\", \"spent\": spent}))\n            break\n        s = await step(c)\n        ok = [x for x in s if x[\"ok\"] and x[\"ttft\"]]\n        dur = (max(x[\"t\"] for x in s) - min(x[\"t\"] for x in s)) if s else 0\n        print(json.dumps({\n            \"concurrency\": c,\n            \"rps\":         round(len(ok) \/ dur, 2) if dur else 0,\n            \"tok_per_s\":   round(sum(x[\"tokens\"] for x in ok) \/ dur, 1) if dur else 0,\n            \"err_rate\":    round(1 - len(ok) \/ max(1, len(s)), 4),\n            \"ttft_p50\":    round(pct([x[\"ttft\"] for x in ok], .50), 3),\n            \"ttft_p95\":    round(pct([x[\"ttft\"] for x in ok], .95), 3),\n            \"e2e_p95\":     round(pct([x[\"e2e\"] for x in ok], .95), 3),\n            \"tpot_p95\":    round(pct([(x[\"e2e\"] - x[\"ttft\"]) \/ max(1, x[\"tokens\"] - 1)\n                                      for x in ok], .95), 4),\n        }))\n\nasyncio.run(main())\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two details are deliberate. The pool is sized to the step&#8217;s concurrency \u2014 with <code>httpx<\/code>&#8216;s default of 100, requests past it queue locally and you benchmark your own client. The budget guard exists because a mis-set <code>max_tokens<\/code> is the most common way a load test becomes an unexpected invoice.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Measuring streaming vs non-streaming correctly<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">TTFT is only observable in streaming mode. In a non-streaming call the response arrives as one buffered JSON body, so first byte is last byte and TTFT degenerates to end-to-end. You lose the split between queue-and-prefill cost and decode cost \u2014 exactly what you need to decide whether to shorten prompts or change models.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Define TTFT explicitly and keep the definition fixed.<\/strong> Most providers send an initial delta carrying only the role and no content. Timestamping the first SSE frame measures network arrival; the first frame with non-empty content measures time to first real token. They differ by tens of milliseconds \u2014 pick one and use it in every run, or your numbers are not comparable.<\/li>\n<li><strong>Check for buffering between you and the provider.<\/strong> A reverse proxy with response buffering, or a CDN in front of your API, coalesces chunks and destroys TTFT as a signal \u2014 you measure your own proxy&#8217;s flush behaviour instead. The tell is one large chunk instead of a stream of small ones; our guide to <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-streaming-sse\/\">AI API streaming<\/a> covers the wire format and this failure mode.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Parse raw SSE frames rather than a client that aggregates the stream for you \u2014 aggregation hands you a complete message and silently makes TTFT unmeasurable. Reuse connections too: without keep-alive you time TCP and TLS setup on every request.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Finding the knee<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Plot throughput in output tokens per second and p95 TTFT against concurrency. Throughput climbs roughly linearly, then plateaus; TTFT sits flat, then bends upward. <strong>The knee is the last step before p95 TTFT exceeds about 1.5\u00d7 its low-concurrency baseline<\/strong>, or before the error rate crosses 0.1%. Past it you add load without adding capacity, degrading everyone already in flight.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Three signatures, three owners:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>429s appear.<\/strong> You crossed a provider request- or token-per-minute ceiling. The fix is quota, key distribution, or request shaping \u2014 see <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-rate-limits-429-errors\/\">rate limits<\/a> for retry and backoff patterns that survive it.<\/li>\n<li><strong>No errors, TTFT flat, throughput plateaus.<\/strong> The provider is batching harder and your tokens per second are capped; TPOT rising while TTFT holds steady is the fingerprint. You need more capacity or a smaller tier.<\/li>\n<li><strong>Latency grows with zero errors and a healthy provider.<\/strong> Almost always your own client: a connection pool smaller than your concurrency, synchronous code blocking an async event loop, or DNS resolution on the request path.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Watch the framework defaults that quietly cap you: <code>httpx<\/code> defaults to 100 connections, a <code>requests.Session<\/code> keeps roughly 10 per host, and several Node HTTP agents disable keep-alive. Pass any of those without an explicit pool size and the knee you found is your client&#8217;s, not the provider&#8217;s.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One trick separates the two cleanly. Run a <strong>provider queue probe<\/strong> alongside the ramp: on a separate connection, every five seconds send a trivial prompt with <code>max_tokens: 1<\/code>. Its TTFT is essentially queue wait plus a tiny prefill, with almost no decode. If the probe rises in lockstep with the main test, the provider is queueing; if it stays flat while your p95 climbs, the bottleneck is yours. Test at your production hour and region too \u2014 a clean ramp at 03:00 UTC says nothing about your 14:00 UTC peak.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The cost of load testing itself<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Estimate the bill before you run. output tokens \u2248 \u03a3 over steps [ concurrency \u00d7 step_seconds \u00f7 (mean_end_to_end + think_time) \u00d7 mean_output_tokens ]<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Worked example: a 1, 2, 4, 8, 16, 32, 64 ramp with a 60-second hold, 6-second mean end-to-end, 2 seconds of think time, and 400 output tokens per response. Each in-flight slot completes 60 \u00f7 8 = 7.5 requests per step, so the final step alone emits 64 \u00d7 7.5 \u00d7 400 \u2248 192,000 output tokens and the ramp sums to roughly 380,000. Pocket change on a small\/fast tier; worth approving in advance on a frontier tier.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Four ways to cap it without weakening the test:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Find the infrastructure knee on the cheap tier.<\/strong> Connection pool limits, event-loop blocking, and TLS overhead are largely model-independent, so ramping to your target concurrency on the cheapest model with <code>max_tokens<\/code> capped at 64\u2013128 finds your client knee cheaply.<\/li>\n<li><strong>Confirm on the real model, briefly.<\/strong> Once the client knee is known, run two or three steps at and just below it on the model you ship, to calibrate TTFT and TPOT.<\/li>\n<li><strong>Put a hard budget guard in the harness.<\/strong> The <code>BUDGET_TOKENS<\/code> check above is a cumulative counter that aborts the ramp, and must not depend on a billing API being reachable.<\/li>\n<li><strong>Use a dedicated key.<\/strong> Load-test traffic on a production credential consumes quota real users depend on and fires alerts on the wrong dashboard; tag it so it can be excluded from analytics.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The tradeoff: a cheap model finds your client bottleneck but says nothing trustworthy about real TTFT or TPOT, which depend on model size and provider batching \u2014 use it for plumbing, not latency budgets. The levers that make production cheaper make testing cheaper too: see <a href=\"https:\/\/qoraapi.com\/blog\/reduce-ai-api-costs\/\">reducing AI API costs<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Interpreting results and planning capacity<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Your headline capacity number should be <strong>sustained output tokens per second at the knee<\/strong>, not requests per second. RPS shifts with your prompt mix and output lengths; tokens per second is what your provider actually meters.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Convert demand to capacity with Little&#8217;s Law. At 3 requests per second and 6-second mean end-to-end, you need 18 generations in flight just to keep up. Safe concurrency must exceed that with margin, which is why the working figure is <strong>knee concurrency \u00d7 0.7<\/strong> \u2014 the rest absorbs bursts and the provider&#8217;s bad hours. Then instances = ceil(peak_in_flight \u00f7 (knee_per_instance \u00d7 0.7)).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In production, alert on leading indicators, not availability. These fire while you still have room to act:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>p95 TTFT above your SLO.<\/strong> The earliest signal that provider-side queueing has started \u2014 it moves before error rates do.<\/li>\n<li><strong>p95 TPOT up more than ~1.5\u00d7 baseline at constant concurrency.<\/strong> The provider raised batching pressure; your effective capacity shrank with no change on your side.<\/li>\n<li><strong>429 rate above 0.1%.<\/strong> Not zero \u2014 a trickle is normal under bursty traffic and should be absorbed by retry with backoff.<\/li>\n<li><strong>Tokens per second per instance falling, or cost per request drifting upward.<\/strong> Throughput per unit of load is degrading, or output length is creeping.<\/li>\n<li><strong>Goodput ratio falling.<\/strong> Requests still succeed, but fewer meet both SLOs \u2014 the honest measure of usability.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Two habits keep them meaningful. Re-run the ramp monthly and after any provider model change \u2014 a silent model swap can move TPOT and TTFT with zero code changes. And log TTFT, TPOT, and token counts per request rather than per aggregate, so the dashboards behind these alerts have percentiles to compute \u2014 see <a href=\"https:\/\/qoraapi.com\/blog\/llm-observability\/\">LLM observability<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you would rather not maintain per-provider clients, pool sizing, and retry logic yourself, an OpenAI-compatible relay puts one endpoint in front of many models \u2014 the same harness ramps a different tier by changing one string, and a provider slowdown can be routed around instead of absorbed. That is what <a href=\"https:\/\/qoraapi.com\/\" target=\"_blank\" rel=\"noopener\">qoraapi.com<\/a> provides: one base URL and key across many models.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">How many concurrent requests can my LLM app handle?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Run the stepped ramp and read it off the chart: it is the last concurrency step before p95 TTFT bends upward or the error rate crosses 0.1%, multiplied by 0.7 for margin. Report it as sustained output tokens per second rather than a request count, because capacity depends on how long each generation runs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Why is my TTFT high while my CPU is nearly idle?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Because the queue you are waiting in belongs to the provider. TTFT is dominated by provider-side queue wait plus prefill of your input, so local CPU, memory, and network metrics stay low while latency climbs. Confirm it with a parallel probe carrying a trivial prompt and <code>max_tokens: 1<\/code> \u2014 if its TTFT rises with the main test, the delay is provider-side.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should I load test with streaming or non-streaming requests?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Use whichever mode you ship; if you stream in production, test with streaming. Streaming is the only mode where TTFT is observable \u2014 a non-streaming response arrives as one buffered body, so time to first byte equals end-to-end and you cannot separate prefill cost from decode cost.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How long should each concurrency step hold?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Sixty to one hundred twenty seconds per step, discarding the first 15\u201330 seconds as warmup. Shorter holds measure connection pool filling rather than steady state, which makes early steps look artificially slow and can hide a knee that only appears once the provider&#8217;s queue builds up.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">LLM load testing is a measurement problem before it is a tooling problem. Sample prompts from the production length distribution so caching does not flatter you, stream every request so TTFT is observable, ramp in held steps so you can see the bend, and separate the provider&#8217;s queue from your own client pool with a parallel probe. Then express capacity as sustained output tokens per second at the knee, keep 30% headroom, and alert on p95 TTFT and p95 TPOT.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Do that and you will know, before your users do, how much load your LLM feature can take \u2014 and what it costs to serve each request.<\/p>\n\n\n\n\n<h3 class=\"wp-block-heading\">Related reading<\/h3>\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-api-rate-limits-429-errors\/\">How to Handle AI API Rate Limits and 429 Errors<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-api-streaming-sse\/\">AI API Streaming Explained: How SSE Works and How to Consume It<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/llm-observability\/\">LLM Observability: Monitoring AI API Usage, Latency and Cost<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-api-failover-multi-provider\/\">How to Build a Multi-Provider AI Failover Layer for 99.9% Uptime<\/a><\/li><\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>LLM load testing is not REST load testing. Measure TTFT, inter-token latency, and cost per request, ramp concurrency realistically, and find your real knee.<\/p>\n","protected":false},"author":1,"featured_media":165,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,6,9,7],"class_list":["post-166","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-api","tag-ai-api","tag-api-gateway","tag-developer-tools","tag-developers"],"_links":{"self":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/comments?post=166"}],"version-history":[{"count":1,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/166\/revisions"}],"predecessor-version":[{"id":178,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/166\/revisions\/178"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media\/165"}],"wp:attachment":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media?parent=166"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/categories?post=166"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/tags?post=166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}