Load testing an LLM app means ramping concurrency in steps while recording time to first token (TTFT), inter-token latency, throughput in tokens per second, and error rate — then finding the concurrency at which p95 TTFT stops being flat. REST-style RPS testing misses this entirely, because LLM latency and cost scale with generated tokens, not with requests.
Why LLM load testing is different from REST load testing
A REST handler’s service time is roughly constant and independent of payload, so you size capacity in requests per second and latency stays flat until a resource saturates. LLM endpoints break that model in four ways.
- Output length is a random variable. A REST handler returns a fixed-size row; an LLM returns however many tokens it decides to emit. Since end-to-end latency ≈ TTFT + tokens × per-token time, latency and cost are both random variables. One fixed prompt samples a distribution and says nothing about p95.
- Streaming holds the connection open for the whole generation. A REST call occupies a worker for milliseconds; a streaming generation holds an in-flight slot for 4–20 seconds. Concurrency here means concurrent generations, governed by Little’s Law: in-flight = arrival rate × mean service time.
- The provider queues on your behalf. Your process can sit at 5% CPU while p95 TTFT triples, because the queue is on someone else’s infrastructure. Local resource metrics are useless as a saturation signal; the only honest instrument is client-side timing.
- Load tests have an invoice. Cost scales with tokens generated, so a test emitting 10× more output tokens costs 10× more — well designed or not.
Provider limits are usually enforced per key on both requests per minute and tokens per minute, so testing on the credential that serves live users trips the ceiling for real traffic — use a separate key, and see our guide to rate limits. The upshot: capacity extrapolated from a single-prompt, non-streaming, low-concurrency test is wrong in the optimistic direction — the dangerous direction.
The metrics that actually matter
Six metrics carry almost all the information. Measure them per request on the client, then aggregate per concurrency step — never as one mean.
| Metric | Precise definition | How to measure it | What it diagnoses |
|---|---|---|---|
| TTFT (time to first token) | Send to first chunk with non-empty content | Timestamp before the HTTP call; timestamp the first SSE delta with content | Provider queueing + prefill |
| TPOT / inter-token latency | (end-to-end − TTFT) ÷ (output_tokens − 1) | Derive per request from the two timestamps and the token count | Decode speed; rises when the provider batches harder |
| End-to-end latency | Send to final token | Client-side timer around the whole stream | Batch-job and non-streaming UX |
| Throughput | Output tokens ÷ wall-clock seconds of the step | Aggregate over the hold window | Capacity. Better than RPS, which is not portable across prompt mixes |
| Goodput | Requests meeting both the TTFT and end-to-end SLOs | Count per step against your SLO thresholds | Usable capacity — high throughput with blown TTFT is not shippable |
| Cost per request | (input_tokens × input_ratio) + (output_tokens × output_ratio) | Token counters from the API, weighted by relative tier pricing | Test-budget predictability; output-length drift |
Two rules make these numbers trustworthy. First, report percentiles, never means: LLM latency distributions are heavy-tailed, and a few requests stuck behind a provider queue drag a mean around. Report p50, p95, and p99.
Second, never collapse TTFT into end-to-end. TTFT is dominated by queue wait plus prefill of your input; the remainder by decode. Halving your system prompt improves TTFT and leaves TPOT untouched; a provider raising its batch size does the reverse.
Designing a realistic load test
1. Sample the prompt mix from production, weighted by traffic share. Bucket real inputs by input-token count — short under 200, medium 200–1500, long over 1500 — and weight each bucket by its traffic share, using at least 50 distinct prompts per bucket or a random nonce. Replaying one identical prompt thousands of times triggers prompt caching, so you measure an artificially fast, artificially cheap system that does not exist for real users.
2. Cap max_tokens at your production cap. Uncapped outputs turn a 60-second step into a multi-thousand-token step and blow the budget. The cap is what production uses, so it belongs in the test.
3. Model think time, and know whether you are closed-loop or open-loop. A closed-loop harness with a fixed worker count self-throttles: as latency rises, each worker completes fewer requests, offered load silently drops, and you under-report queueing. Finding the true knee needs an open-loop run at a fixed arrival rate, where offered load stays constant while latency grows.
4. Ramp in steps and discard warmup. Use a geometric ramp (1, 2, 4, 8, 16, 32, 64), hold each step 60–120 seconds, and discard the first 15–30 seconds. Shorter holds measure connection pool filling, not steady state.
This harness ramps concurrency, streams every request, records TTFT per request, and prints a percentile summary per step. Install with pip install httpx.
import asyncio, json, random, time
import httpx
URL = "https://your-gateway/v1/chat/completions"
KEY = "sk-..." # a dedicated load-test key, NOT the production one
MODEL = "cheap-small-tier-model" # ramp on the cheap tier, confirm on the real one
# (weight, prompt, max_tokens) sampled from the production length distribution
MIX = [
(0.60, "Classify the sentiment of this review: ...", 32),
(0.30, "Summarize this support ticket in three bullets: ...", 200),
(0.10, "Extract every line item into JSON: ...", 700),
]
RAMP, HOLD_S, WARMUP_S, THINK_S = [1, 2, 4, 8, 16, 32, 64], 60, 15, 2.0
BUDGET_TOKENS = 400_000 # hard stop so the test cannot run away
spent = 0
def sample(rng):
r, acc = rng.random(), 0.0
for w, text, mt in MIX:
acc += w
if r <= acc:
return text, mt
return MIX[-1][1], MIX[-1][2]
async def one(client, rng, rec):
global spent
text, max_tokens = sample(rng)
body = {"model": MODEL, "stream": True, "max_tokens": max_tokens,
"messages": [{"role": "user", "content": text}]}
t0 = time.perf_counter()
ttft = None
toks = 0
try:
async with client.stream("POST", URL, json=body,
headers={"Authorization": f"Bearer {KEY}"}) as r:
r.raise_for_status()
async for line in r.aiter_lines():
if not line.startswith("data:"):
continue
chunk = line[5:].strip()
if chunk == "[DONE]":
break
delta = json.loads(chunk)["choices"][0].get("delta", {})
if delta.get("content"):
toks += 1
if ttft is None:
ttft = time.perf_counter() - t0 # first CONTENT token
rec.append({"ok": True, "ttft": ttft, "e2e": time.perf_counter() - t0,
"tokens": toks, "t": time.perf_counter()})
spent += toks
except Exception as e:
rec.append({"ok": False, "err": type(e).__name__, "t": time.perf_counter()})
async def worker(client, rng, rec, stop):
while not stop.is_set():
await one(client, rng, rec)
await asyncio.sleep(rng.expovariate(1 / THINK_S)) # user think time
async def step(concurrency):
rec, stop, rng = [], asyncio.Event(), random.Random(42)
limits = httpx.Limits(max_connections=concurrency,
max_keepalive_connections=concurrency)
async with httpx.AsyncClient(timeout=120, limits=limits) as client:
tasks = [asyncio.create_task(worker(client, rng, rec, stop))
for _ in range(concurrency)]
await asyncio.sleep(WARMUP_S)
cut = len(rec) # discard warmup samples
await asyncio.sleep(HOLD_S)
stop.set()
await asyncio.gather(*tasks, return_exceptions=True)
return rec[cut:]
def pct(xs, p):
xs = sorted(xs)
return xs[min(len(xs) - 1, int(len(xs) * p))] if xs else float("nan")
async def main():
for c in RAMP:
if spent > BUDGET_TOKENS:
print(json.dumps({"aborted": "token budget exhausted", "spent": spent}))
break
s = await step(c)
ok = [x for x in s if x["ok"] and x["ttft"]]
dur = (max(x["t"] for x in s) - min(x["t"] for x in s)) if s else 0
print(json.dumps({
"concurrency": c,
"rps": round(len(ok) / dur, 2) if dur else 0,
"tok_per_s": round(sum(x["tokens"] for x in ok) / dur, 1) if dur else 0,
"err_rate": round(1 - len(ok) / max(1, len(s)), 4),
"ttft_p50": round(pct([x["ttft"] for x in ok], .50), 3),
"ttft_p95": round(pct([x["ttft"] for x in ok], .95), 3),
"e2e_p95": round(pct([x["e2e"] for x in ok], .95), 3),
"tpot_p95": round(pct([(x["e2e"] - x["ttft"]) / max(1, x["tokens"] - 1)
for x in ok], .95), 4),
}))
asyncio.run(main())
Two details are deliberate. The pool is sized to the step’s concurrency — with httpx‘s default of 100, requests past it queue locally and you benchmark your own client. The budget guard exists because a mis-set max_tokens is the most common way a load test becomes an unexpected invoice.
Measuring streaming vs non-streaming correctly
TTFT is only observable in streaming mode. In a non-streaming call the response arrives as one buffered JSON body, so first byte is last byte and TTFT degenerates to end-to-end. You lose the split between queue-and-prefill cost and decode cost — exactly what you need to decide whether to shorten prompts or change models.
- Define TTFT explicitly and keep the definition fixed. Most providers send an initial delta carrying only the role and no content. Timestamping the first SSE frame measures network arrival; the first frame with non-empty content measures time to first real token. They differ by tens of milliseconds — pick one and use it in every run, or your numbers are not comparable.
- Check for buffering between you and the provider. A reverse proxy with response buffering, or a CDN in front of your API, coalesces chunks and destroys TTFT as a signal — you measure your own proxy’s flush behaviour instead. The tell is one large chunk instead of a stream of small ones; our guide to AI API streaming covers the wire format and this failure mode.
Parse raw SSE frames rather than a client that aggregates the stream for you — aggregation hands you a complete message and silently makes TTFT unmeasurable. Reuse connections too: without keep-alive you time TCP and TLS setup on every request.
Finding the knee
Plot throughput in output tokens per second and p95 TTFT against concurrency. Throughput climbs roughly linearly, then plateaus; TTFT sits flat, then bends upward. The knee is the last step before p95 TTFT exceeds about 1.5× its low-concurrency baseline, or before the error rate crosses 0.1%. Past it you add load without adding capacity, degrading everyone already in flight.
Three signatures, three owners:
- 429s appear. You crossed a provider request- or token-per-minute ceiling. The fix is quota, key distribution, or request shaping — see rate limits for retry and backoff patterns that survive it.
- No errors, TTFT flat, throughput plateaus. The provider is batching harder and your tokens per second are capped; TPOT rising while TTFT holds steady is the fingerprint. You need more capacity or a smaller tier.
- Latency grows with zero errors and a healthy provider. Almost always your own client: a connection pool smaller than your concurrency, synchronous code blocking an async event loop, or DNS resolution on the request path.
Watch the framework defaults that quietly cap you: httpx defaults to 100 connections, a requests.Session keeps roughly 10 per host, and several Node HTTP agents disable keep-alive. Pass any of those without an explicit pool size and the knee you found is your client’s, not the provider’s.
One trick separates the two cleanly. Run a provider queue probe alongside the ramp: on a separate connection, every five seconds send a trivial prompt with max_tokens: 1. Its TTFT is essentially queue wait plus a tiny prefill, with almost no decode. If the probe rises in lockstep with the main test, the provider is queueing; if it stays flat while your p95 climbs, the bottleneck is yours. Test at your production hour and region too — a clean ramp at 03:00 UTC says nothing about your 14:00 UTC peak.
The cost of load testing itself
Estimate the bill before you run. output tokens ≈ Σ over steps [ concurrency × step_seconds ÷ (mean_end_to_end + think_time) × mean_output_tokens ]
Worked example: a 1, 2, 4, 8, 16, 32, 64 ramp with a 60-second hold, 6-second mean end-to-end, 2 seconds of think time, and 400 output tokens per response. Each in-flight slot completes 60 ÷ 8 = 7.5 requests per step, so the final step alone emits 64 × 7.5 × 400 ≈ 192,000 output tokens and the ramp sums to roughly 380,000. Pocket change on a small/fast tier; worth approving in advance on a frontier tier.
Four ways to cap it without weakening the test:
- Find the infrastructure knee on the cheap tier. Connection pool limits, event-loop blocking, and TLS overhead are largely model-independent, so ramping to your target concurrency on the cheapest model with
max_tokenscapped at 64–128 finds your client knee cheaply. - Confirm on the real model, briefly. Once the client knee is known, run two or three steps at and just below it on the model you ship, to calibrate TTFT and TPOT.
- Put a hard budget guard in the harness. The
BUDGET_TOKENScheck above is a cumulative counter that aborts the ramp, and must not depend on a billing API being reachable. - Use a dedicated key. Load-test traffic on a production credential consumes quota real users depend on and fires alerts on the wrong dashboard; tag it so it can be excluded from analytics.
The tradeoff: a cheap model finds your client bottleneck but says nothing trustworthy about real TTFT or TPOT, which depend on model size and provider batching — use it for plumbing, not latency budgets. The levers that make production cheaper make testing cheaper too: see reducing AI API costs.
Interpreting results and planning capacity
Your headline capacity number should be sustained output tokens per second at the knee, not requests per second. RPS shifts with your prompt mix and output lengths; tokens per second is what your provider actually meters.
Convert demand to capacity with Little’s Law. At 3 requests per second and 6-second mean end-to-end, you need 18 generations in flight just to keep up. Safe concurrency must exceed that with margin, which is why the working figure is knee concurrency × 0.7 — the rest absorbs bursts and the provider’s bad hours. Then instances = ceil(peak_in_flight ÷ (knee_per_instance × 0.7)).
In production, alert on leading indicators, not availability. These fire while you still have room to act:
- p95 TTFT above your SLO. The earliest signal that provider-side queueing has started — it moves before error rates do.
- p95 TPOT up more than ~1.5× baseline at constant concurrency. The provider raised batching pressure; your effective capacity shrank with no change on your side.
- 429 rate above 0.1%. Not zero — a trickle is normal under bursty traffic and should be absorbed by retry with backoff.
- Tokens per second per instance falling, or cost per request drifting upward. Throughput per unit of load is degrading, or output length is creeping.
- Goodput ratio falling. Requests still succeed, but fewer meet both SLOs — the honest measure of usability.
Two habits keep them meaningful. Re-run the ramp monthly and after any provider model change — a silent model swap can move TPOT and TTFT with zero code changes. And log TTFT, TPOT, and token counts per request rather than per aggregate, so the dashboards behind these alerts have percentiles to compute — see LLM observability.
If you would rather not maintain per-provider clients, pool sizing, and retry logic yourself, an OpenAI-compatible relay puts one endpoint in front of many models — the same harness ramps a different tier by changing one string, and a provider slowdown can be routed around instead of absorbed. That is what qoraapi.com provides: one base URL and key across many models.
Frequently asked questions
How many concurrent requests can my LLM app handle?
Run the stepped ramp and read it off the chart: it is the last concurrency step before p95 TTFT bends upward or the error rate crosses 0.1%, multiplied by 0.7 for margin. Report it as sustained output tokens per second rather than a request count, because capacity depends on how long each generation runs.
Why is my TTFT high while my CPU is nearly idle?
Because the queue you are waiting in belongs to the provider. TTFT is dominated by provider-side queue wait plus prefill of your input, so local CPU, memory, and network metrics stay low while latency climbs. Confirm it with a parallel probe carrying a trivial prompt and max_tokens: 1 — if its TTFT rises with the main test, the delay is provider-side.
Should I load test with streaming or non-streaming requests?
Use whichever mode you ship; if you stream in production, test with streaming. Streaming is the only mode where TTFT is observable — a non-streaming response arrives as one buffered body, so time to first byte equals end-to-end and you cannot separate prefill cost from decode cost.
How long should each concurrency step hold?
Sixty to one hundred twenty seconds per step, discarding the first 15–30 seconds as warmup. Shorter holds measure connection pool filling rather than steady state, which makes early steps look artificially slow and can hide a knee that only appears once the provider’s queue builds up.
Conclusion
LLM load testing is a measurement problem before it is a tooling problem. Sample prompts from the production length distribution so caching does not flatter you, stream every request so TTFT is observable, ramp in held steps so you can see the bend, and separate the provider’s queue from your own client pool with a parallel probe. Then express capacity as sustained output tokens per second at the knee, keep 30% headroom, and alert on p95 TTFT and p95 TPOT.
Do that and you will know, before your users do, how much load your LLM feature can take — and what it costs to serve each request.
Leave a Reply