An A/B test is the only mechanism that reliably tells you whether a prompt edit or a model swap made your product better, and most teams run them badly enough that the result is worse than no test at all. If your feature calls an LLM, the prompt is production code with no type checker, no unit test covering real inputs, and a behaviour that shifts when a provider updates a model behind the same alias. The experiment is your regression suite.
Why an LLM change needs an experiment at all
A prompt edit is a code change with no compiler. Change “Summarise the ticket” to “Summarise the ticket in one paragraph, no bullet points” and every unit test still passes, because tests assert on JSON shape and field presence, not on whether the summary still preserves the escalation reason. Nothing in CI can see that regression.
Offline evals do not predict production
Eval sets are written by the people who wrote the prompt, from the cases they had in mind. Production traffic is long-tailed: pasted stack traces, mixed-language input, empty strings, 40,000-token documents, users typing “no” into a date field. A 200-example eval set proves you did not break the cases you already considered, and says nothing about the strange 3% where the support tickets come from. Run the offline eval harness as a cheap filter, then experiment on the survivors.
Provider-side changes are unannounced
Model aliases are pointers. A dated snapshot is safer than a floating alias, but even pinned snapshots get silent infrastructure and quantisation changes. Log the version returned in the response, not the alias you requested, or you will attribute a provider-side change to your own prompt edit.
What is actually random here
Three sources of variance, and conflating them is the most common design error. Model non-determinism covers sampling above temperature zero plus batch-size-dependent kernels and mixture-of-experts routing: temperature 0 removes sampling noise but not byte-identical output across calls, let alone across versions. Traffic mix is who arrives during the window, and in most real experiments it dominates. Assignment maps a unit to a variant, and becomes a noise source rather than a control if it is not deterministic and stable.
So never let the model be both the treatment and the source of noise. If arms differ in temperature as well as in prompt, the measured difference mixes the prompt effect with sampling variance, and one request per condition cannot separate them. Fix temperature, top_p and seed across arms; vary exactly one thing.
Unit of randomisation: user, session, or request
The unit decides whether you measure a user-visible effect or a statistical artefact.
| Unit | Assignment key | Contamination risk | Statistical power | Use when |
|---|---|---|---|---|
| Request | request_id | High: one user sees both behaviours inside a session | Highest, because units are plentiful | Internal batch jobs nobody reads, genuinely independent requests |
| Session | session_id | Medium: consistent within a conversation, not across them | Medium | A session is the entire unit of value and there is no account |
| User or tenant | user_id, tenant_id | None | Lowest, because correlated outcomes shrink effective n | Anything a human sees, anything with memory, anything touching retention |
Per-request assignment contaminates the experience: the same user watches the assistant answer in one style and then another, with different refusals and different latency. Worse, the user’s behaviour then depends on the mix of variants they received, which breaks the comparison you wanted.
Per-session assignment looks like a middle ground and is mostly a trap. Sessions are short, correlated and interleaved for the same user, so you still get visible inconsistency and still have to cluster variance at the user level. If you are clustering at the user level anyway, you had user-level power all along.
Recommendation: randomise by user, or by tenant for B2B products. The power cost is real, and the section below quantifies it.
Sticky assignment: hash, do not store
Assignment should be a pure function of experiment id, unit id and salt. No storage, no cookie, no database read on the hot path.
import hashlib
import struct
def assign_variant(experiment_id: str, unit_id: str,
weights: dict, salt: str = "v1") -> str:
"""Deterministic variant assignment.
The same (experiment_id, unit_id, salt) maps to the same variant on every
service, in every language, forever. No state to read, no state to lose.
"""
total = sum(weights.values())
key = "{0}:{1}:{2}".format(experiment_id, unit_id, salt).encode("utf-8")
# First 8 bytes of SHA-256 as an unsigned big-endian integer, bucketed
# into [0, total). SHA-256 is used as a hash here, not for security.
bucket = struct.unpack(">Q", hashlib.sha256(key).digest()[:8])[0] % total
cursor = 0
for variant, weight in weights.items():
cursor += weight
if bucket < cursor:
return variant
raise AssertionError("unreachable: bucket out of range")
# 90 / 10 canary split. Changing the weights REQUIRES a new salt, otherwise
# users already measured get silently reassigned and both arms are corrupted.
weights = {"control": 90, "candidate": 10}
variant = assign_variant("checkout-summary-v3", user_id, weights, salt="v1")
Why not a cookie alone: cookies are mutable, cleared, per-device and increasingly blocked. A user who clears cookies gets re-randomised, and those users are not a random sample, so you have injected selection bias into your assignment. Persist it server-side for audit, but derive it from the hash.
Use a different salt per experiment, or the same users land in bucket zero for every test and your portfolio-wide control group becomes a fixed, non-random subset. Freeze the salt at launch: re-weighting a running experiment silently reassigns users you have already measured.
Metrics: one primary, everything else is a guardrail
| Class | Metric | Definition | Failure it catches |
|---|---|---|---|
| Primary | Task success | Thumbs up, ticket resolved, suggestion accepted, downstream conversion | Whether the change helped at all |
| Guardrail | p95 latency | End-to-end including retries | Perceived slowness, client timeouts |
| Guardrail | Cost per successful task | Total spend divided by successes | A cheap model that retries its way back to expensive |
| Guardrail | Refusal and empty-response rate | Responses that decline or return nothing usable | Over-aligned prompt, truncated context |
| Guardrail | Error and retry rate | 5xx, timeouts, schema validation failures | Provider breakage, parser drift |
| Guardrail | Safety incidents | Policy violations, PII leakage, unsafe tool calls | Compliance exposure |
| Diagnostic | Tokens, cache hit rate, tool calls | Not decision metrics | Explaining a move, not deciding one |
There should be exactly one primary metric; three means none, because you will report whichever moved.
Cost and latency are guardrails, not tie-breakers. Pre-register non-inferiority bounds: the candidate may not raise p95 latency by more than 15%, cost per successful task by more than 5%, or the refusal rate by more than two percentage points. A breach is a loss regardless of the primary metric, because a quality win that triples the inference bill is a product decision rather than a test result.
Emit diagnostics on the same event stream as the primary metric, so attribution does not require joining three systems later. That is what makes LLM observability useful rather than decorative.
Measuring quality when quality is not a number
Pairwise preference beats absolute scoring
Ask raters to pick between two outputs for the same input rather than score one from 1 to 5, because absolute scales drift between raters and within a rater across a session. Randomise left and right order to control position bias, blind the rater to variant identity, and measure agreement first: if two humans agree 55% of the time, the metric is noise. An LLM judge needs validating against human labels on a held-out sample, because a judge from the candidate’s own model family prefers its own outputs.
Implicit signals, and the proxy trap
Every request produces traces without asking: copy-to-clipboard, acceptance of a suggestion, edit distance between suggested and sent text, regeneration, abandonment, escalation to a human. They arrive at request scale, so they reach significance far sooner than a thumbs-up that 2% of users click. They are also gameable: optimising copy rate rewards long outputs, optimising edit distance rewards outputs that resemble the user’s draft, optimising retry rate rewards a model that is confidently wrong. Anchor at least one metric to something the business already counts, such as revenue or time-to-close: a dense proxy plus a sparse real outcome is workable, a dense proxy alone is a metric you will optimise into absurdity.
Sample size: the arithmetic that kills most experiments
For a two-arm test on a proportion, the sample per arm is n = (z_alpha + z_beta)^2 * (p1(1-p1) + p2(1-p2)) / (p1-p2)^2. At alpha 0.05 two-sided and 80% power the z values are 1.96 and 0.84, so the squared sum is 7.84. Take a baseline task-success rate of 12%.
- Detect a 10% relative lift, 12% to 13.2%: p1(1-p1) = 0.1056, p2(1-p2) = 0.1146, sum 0.2202, and (p1-p2)^2 = 0.000144. n = 7.84 x 0.2202 / 0.000144 = 11,990 per arm.
- Detect a 5% relative lift, 12% to 12.6%: sum 0.2157, (p1-p2)^2 = 0.000036, so n = 46,970 per arm.
- Detect a 2% relative lift, 12% to 12.24%: sum 0.2130, (p1-p2)^2 = 0.00000576, so n = 289,900 per arm.
Those are independent observations, and users are not independent of each other. At 20 requests per user and an intra-user correlation of 0.3, the design effect is 1 + 19 x 0.3 = 6.7. Reaching the equivalent of 11,990 independent observations needs roughly 11,990 x 6.7 = 80,000 requests per arm, or about 4,000 users. The same multiplier on the 2% lift means 1.94 million requests per arm, 3.9 million across both arms: at 50,000 requests a day, 78 days.
You cannot detect a 2% relative quality win at that traffic. Not “it is hard”: you cannot, and a test that reports significance there reports noise. Decide the minimum detectable effect before launch, and if the arithmetic says 78 days, do not run the test. Ship behind a canary with guardrails, reduce the noise in the metric, or accept the change on qualitative grounds and say so out loud. Extra arms make this worse: each one needs its own full sample, and with k arms the family-wise error rate at alpha 0.05 is 1 – 0.95^(k-1), which is 14% at four arms.
Peeking, novelty, and the calendar
The peeking problem
Stop the first time p drops below 0.05 and your false-positive rate is not 5%; it is much higher, approaching 1 with enough looks. Under the null your test statistic is a random walk, and the chance it crosses a fixed boundary before your planned sample size far exceeds nominal alpha. A fixed-horizon test guarantees 5% only if you look once, at the pre-registered n.
Two legitimate fixes. A fixed horizon: freeze the sample size and analysis, look once, decide. If you need interim monitoring, use group-sequential boundaries with alpha spending, where O’Brien-Fleming is standard for a small number of looks. Or design for continuous monitoring from the start, using always-valid confidence sequences or a Bayesian rule with a pre-registered posterior threshold. What is not legitimate is a stopping rule chosen after seeing the data. Guardrail breaches are the exception: stopping for harm is safety, not efficacy.
Novelty, primacy, and seasonal traffic
Users engage more with anything new and habituate to anything that persists, and both effects decay over days. A two-day test measures the novelty spike rather than the steady state you are shipping. Run at least one full week, and prefer two when the metric is user-mediated; mechanical metrics such as latency and cost need no such window.
Do not run an experiment across a holiday, a marketing campaign, a pricing change or a product launch. Traffic mix shifts and the control group stops being a valid counterfactual. If a launch is unavoidable, log it as a covariate and plan to re-run.
Model swaps: cost per successful task, not cost per call
Worked example, 10,000 requests per arm. Incumbent model A costs $0.0040 per call and succeeds 92% of the time with no retries. Candidate B costs $0.0012 per call and succeeds 84% on the first attempt; your client retries 25% of first-attempt failures once, and a retry succeeds 84% of the time at the same price.
- A: cost is 10,000 x $0.0040 = $40.00, successes are 9,200, so cost per success is $40.00 / 9,200 = $0.00435.
- B: first-attempt failures are 1,600, of which 400 are retried. Retry cost is 400 x $0.0012 = $0.48, total cost $12.00 + $0.48 = $12.48. Retry successes are 400 x 0.84 = 336, total successes 8,400 + 336 = 8,736, so cost per success is $12.48 / 8,736 = $0.00143.
B is about three times cheaper per success, so on cost alone it wins. Now value the success: at $0.50 of downstream value per task, A generates 9,200 x $0.50 = $4,600 and B generates 8,736 x $0.50 = $4,368. B saves $27.52 in inference and gives up $232.00 in value. Cost per call hid that.
A slower model can push your p95 past a client timeout, and the resulting retries and abandonment surface as a success-rate drop, because abandoned requests never log a completion. Routing across providers should be a configuration change rather than a deploy, which is the main argument for keeping routing decisions out of application code. Any serious cost reduction programme works the same way: measure per outcome, not per call.
Rollout mechanics: shadow, canary, then experiment
Shadow mode sends a copy of production traffic to the candidate and logs both outputs, serving only the incumbent. Zero user risk, and it validates schema conformance, latency, token counts and cost before anyone sees the output. It cannot tell you whether quality improved, because nobody reads the shadow output: it is a gate, not evidence.
A canary sends 1% to 5% of real traffic to the new variant with automatic rollback on a guardrail breach. It answers “is it safe to expose users to this”, which is a different question from “is it better”, and most prompt changes should stop here.
Under a flag, the variant must still come from the same hash rather than from per-request state. If a service restart reshuffles users between variants, your experiment silently becomes a per-request test with extra steps. Cache the resolved variant against the assignment key, with no TTL short enough to expire mid-session.
Roll back in under a minute, and note that this constrains architecture rather than process. Prompts must live in a versioned store a config write can point at, not inside a container image; a prompt change that needs a deploy is one you will not roll back at 02:00. Keep the previous version warm so rollback is a pointer flip, and put the prompt hash in every cache key, or a rollback will keep serving the reverted version for the whole cache TTL. See prompt management and versioning.
A concrete implementation
One event per LLM call, written where you know the outcome, with the assignment stamped on it. Not two tables joined later.
| Column | Type | Why it is there |
|---|---|---|
| experiment_id | string | Join key, also the salt scope |
| variant | string | Resolved at assignment, never inferred later |
| unit_id | string | User or tenant id, hashed before storage if it is PII |
| unit_type | string | Makes the randomisation unit explicit in the data |
| assignment_salt | string | Lets you discard a bad randomisation without guessing |
| model_requested | string | The alias you asked for |
| model_version | string | The exact version returned, because the alias lies |
| prompt_hash | string | SHA-256 of the rendered prompt, not the prompt itself |
| params | json | temperature, top_p, max_tokens, seed |
| latency_ms | int | End to end, including every retry |
| attempts | int | Anything above 1 is a retry, and a cost event |
| input_tokens, output_tokens | int | Recompute cost when pricing changes |
| cost_usd | decimal | Provider-reported or computed, but always present |
| success | bool | Your task-success signal, not an HTTP 200 |
| guardrail_breach | string array | latency, cost, refusal, safety, schema |
WITH base AS (
SELECT
a.variant,
a.unit_id,
r.success,
r.latency_ms,
r.cost_usd
FROM experiment_assignments AS a
JOIN llm_requests AS r
ON r.unit_id = a.unit_id
AND r.experiment_id = a.experiment_id
AND r.ts >= a.assigned_at
WHERE a.experiment_id = 'checkout-summary-v3'
AND a.unit_type = 'user'
AND r.ts >= TIMESTAMP '2026-09-01'
AND r.ts < TIMESTAMP '2026-09-15'
),
-- One row per USER first. The unit of analysis must match the unit of
-- randomisation, otherwise correlated requests manufacture significance.
per_user AS (
SELECT
variant,
unit_id,
COUNT(*) AS requests,
AVG(CASE WHEN success THEN 1.0 ELSE 0.0 END) AS success_rate,
SUM(CASE WHEN success THEN 0 ELSE 1 END) AS failures,
SUM(cost_usd) AS cost_usd,
APPROX_QUANTILES(latency_ms, 100)[OFFSET(95)] AS user_p95_ms
FROM base
GROUP BY variant, unit_id
)
SELECT
variant,
COUNT(*) AS users,
SUM(requests) AS requests,
AVG(success_rate) AS success_rate,
SUM(cost_usd) / NULLIF(SUM(requests) - SUM(failures), 0) AS cost_per_success_usd,
APPROX_QUANTILES(user_p95_ms, 100)[OFFSET(50)] AS median_user_p95_ms
FROM per_user
GROUP BY variant
ORDER BY variant;The query aggregates to one row per user and only then averages, so the unit of analysis matches the unit of randomisation. Averaging raw requests treats 20 correlated requests from one user as 20 independent observations, which understates variance and manufactures significance; at request level, use cluster-robust standard errors keyed on unit_id. Cost per success is total cost divided by successes, so failed and retried attempts are charged to the successes they eventually produced.
When an A/B test is worth it, and when it is not
Run a real A/B test only when three things hold at once. The effect is user-visible. You have enough traffic to detect the effect size you care about within two weeks. A wrong decision is expensive or hard to reverse. If any fails, ship behind a flag with a canary and a guardrail-driven automatic rollback.
- Ship behind a flag, no A/B test: latency and cost optimisations that provably do not change outputs, provider failover, caching layers, prompt compression verified by output equality on a replay set.
- Run the A/B test: a change to the system prompt of a user-facing assistant, a model swap on a revenue path, a change to how retrieved context is formatted, a change to refusal or escalation policy. These alter what users experience, and you cannot reason your way to the answer.
- Decide without an experiment, and say so: cosmetic changes with no plausible mechanism, and changes where the minimum detectable effect you can reach exceeds the effect you care about.
The failure mode to avoid is the middle: an underpowered test that runs four days, returns p = 0.04 on a secondary metric you never pre-registered, and ships because it has a p-value. That is worse than shipping behind a canary, because it launders a guess into a decision. Put the leverage in the plumbing instead: one gateway key that stamps cost, model version and prompt hash on every event, one assignment function, one event stream. The Qora API gateway is shaped around exactly that: one OpenAI-compatible endpoint with unified billing, failover and cost controls, so a new variant is a routing change rather than an integration project.
Frequently asked questions
Can I A/B test at temperature 0?
Yes, but temperature 0 does not mean deterministic in production. Greedy decoding removes sampling randomness, not batch-size-dependent kernel behaviour, mixture-of-experts routing or provider-side version changes. Identical requests can still return different tokens across calls, and will across model versions. Treat temperature 0 as low variance rather than zero variance, and keep the unit of randomisation at the user level: the variance you are controlling for is mostly traffic, not sampling.
Can I use an LLM as the judge for my primary metric?
Use it as a dense secondary signal after validating it against human labels on a held-out sample; below roughly 0.6 kappa it is too noisy to steer on. Blind the judge to variant identity and randomise candidate order to control position bias, or you are partly measuring self-preference.
What if the primary metric improves but a guardrail breaks?
The variant loses. Guardrails are non-inferiority constraints, not tie-breakers. Pre-register the bounds and treat a breach as a failure to ship, regardless of the p-value on the primary metric.
Conclusion
The decisions that matter are few and all happen before you launch: randomise by user with a hashed, salted, deterministic assignment; pick one primary metric and pre-register the cost and latency guardrails; compute the minimum detectable effect from your real traffic and refuse to run the test if the arithmetic says eleven weeks; fix the horizon; aggregate at the unit of randomisation.
Most teams get the order backwards. They launch an experiment, watch a dashboard, and reach for statistics only when a number looks interesting. The fix is not a better dashboard. It is a deterministic assignment function, an event schema carrying variant, model version and prompt hash on every call, and a rollback path that takes seconds. Build those three things and the experiment becomes cheap enough to run properly.
Leave a Reply