{"id":300,"date":"2026-09-22T17:48:05","date_gmt":"2026-09-22T09:48:05","guid":{"rendered":"https:\/\/wp.qoraapi.com\/llm-eval-harness\/"},"modified":"2026-09-22T17:49:34","modified_gmt":"2026-09-22T09:49:34","slug":"llm-eval-harness","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/llm-eval-harness\/","title":{"rendered":"Building an LLM Eval Harness: Regression Testing for Prompts and Models"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">An LLM eval harness is a versioned set of task-specific cases plus scorers, run on every change to your prompt, model or retrieval stack, that answers one question: did this change break something a user would notice? It is not a leaderboard. MMLU and arena rankings say nothing about whether your invoice extractor still returns the right currency for a scanned PDF from a German vendor.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is the application-level view: scoring your feature, not the model. Public benchmarking is covered in <a href=\"https:\/\/qoraapi.com\/blog\/evaluate-benchmark-ai-models\/\">how to evaluate and benchmark AI models<\/a>; throughput rather than correctness is <a href=\"https:\/\/qoraapi.com\/blog\/load-testing-llm-apps\/\">load testing<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why public benchmarks do not predict your behaviour<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Three reasons, all structural.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Distribution mismatch.<\/strong> Benchmarks sample curated, clean questions. Your traffic is messy: half-paragraph tickets, OCR noise, three languages in one message, a stack trace pasted into a description box. That gap is not a constant you can subtract.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Benchmarks measure the model alone.<\/strong> Your feature is a prompt, retrieved documents, a tool-call loop, a parser and a retry policy. A model that gains points on a reasoning benchmark can make your feature worse, because it grew more verbose and your parser truncates at 2,000 tokens.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Published benchmarks get optimised against.<\/strong> Widely reported numbers become training targets, which makes them non-transferable.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Building the golden set<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Source cases from traffic and incidents<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The best cases are not invented. They come from logs and postmortems.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Sample real inputs, but stratify rather than taking the top hundred by frequency: head traffic already works. Take the long tail \u2014 shortest input, longest input, emoji in the middle, two attachments with a question about the second. Where a router sits upstream, sample branches evenly rather than by traffic share.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Every user-visible incident becomes a permanent case: a ticket saying &#8220;the summary was about the wrong account&#8221; enters the set with an assertion that would have caught it. Never invent cases in a text editor, because you will unconsciously write the ones your prompt already passes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Thirty to a hundred cases to start<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Below 30 cases almost nothing is detectable: moving three of them swings the score by ten points. Above roughly 150, marginal information per case falls fast, because you start adding near-duplicates. The first 50 capture the failure modes you know; the rest buy the second and third standard deviations.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Stratify, weight, and hold something back<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Tag every case on the axes that matter: difficulty, user segment, input shape and risk \u2014 the difference between an answer that is annoying and one that is a compliance problem. A pass rate moving from 91 to 89 percent is a number; a breakdown showing the drop is entirely in the long-input bucket, paid tier only, is a diagnosis.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Weight by risk tag, because a wrong currency code costs more than a verbose sentence. Keep a held-out slice of 20 to 30 percent, picked by a fixed seed and treated as read-only: edit a prompt in response to a failure there and it stops being held out.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Case anatomy<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Input<\/strong> \u2014 verbatim. Preserve whitespace and typos; normalising them changes what you test.<\/li>\n<li><strong>Expected properties<\/strong> \u2014 a dict of assertions, not a string. &#8220;Names the vendor and the total is within one cent of 4210.50&#8221; is a property; &#8220;is exactly this sentence&#8221; is not.<\/li>\n<li><strong>Context and fixtures<\/strong> \u2014 retrieved documents, tool responses, a frozen row. Pin them: an eval that hits a live index turns an index change into an apparent model regression.<\/li>\n<li><strong>Tags<\/strong> \u2014 the stratification axes above, as a flat tuple you can group by.<\/li>\n<li><strong>Weight or risk<\/strong> \u2014 what a failure here costs, so the aggregate means something.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Exact match is usually the wrong assertion<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Generative output has many correct realisations, so asserting the exact string fails on every model upgrade, temperature change and tokenizer whitespace difference. Those failures are noise, and teams end up muting the suite.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Assert properties that must hold for <em>any<\/em> acceptable answer. Summarisation: names all three required entities, under 120 words, no number absent from the source, no competitor mentioned. Classification: the label is in the allowed set. Extraction: the JSON parses, the schema validates, the amount is within tolerance. Exact match survives only where the output space is a small closed enum.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Choosing a scorer<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Match the scorer to the failure you want to detect, not to the one you find easiest to write.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\">\n<thead>\n<tr><th>Scorer<\/th><th>Use it when<\/th><th>Failure mode<\/th><th>Cost per case<\/th><\/tr>\n<\/thead>\n<tbody>\n<tr><td>Exact or regex assertion<\/td><td>The output space is closed: labels, enums, IDs, fixed-format codes<\/td><td>Brittle to any legitimate variation; passes when a wrong answer happens to contain the pattern<\/td><td>CPU, microseconds<\/td><\/tr>\n<tr><td>Programmatic check (parses, schema valid, citation resolves, number in tolerance)<\/td><td>Output is structured, or a property is mechanically checkable<\/td><td>Catches only what you encoded; a valid-but-wrong answer passes<\/td><td>CPU plus any lookup latency<\/td><\/tr>\n<tr><td>Embedding similarity to a reference answer<\/td><td>A reference exists and paraphrases are acceptable, as in semantic search or FAQ answers<\/td><td>High similarity for answers that are confidently wrong on the same topic; the threshold is dataset-specific and needs calibration<\/td><td>One embedding call, cacheable<\/td><\/tr>\n<tr><td>Model-graded with a rubric (LLM-as-judge)<\/td><td>The criterion is subjective or needs reasoning: groundedness, tone, whether a refusal was appropriate<\/td><td>Position, verbosity and self-preference bias; judge noise becomes suite noise<\/td><td>One or more extra generations, often the dominant cost<\/td><\/tr>\n<tr><td>Human review, sampled<\/td><td>Calibrating the judge, adjudicating disagreements, triaging novel failures<\/td><td>Throughput; drift in reviewer standards over weeks<\/td><td>Minutes of human time<\/td><\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Programmatic checks earn more than they look like<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The strongest assertions in production suites are mechanical: the response parses as JSON; required fields are non-empty; every citation ID exists in the retrieved set; every number in the output appears in the context or a tool result; no write tool call precedes a confirmation. These are deterministic, free and fast, and they catch most real regressions. Substring grounding is a crude proxy for hallucination but catches a lot before you pay for a judge \u2014 see <a href=\"https:\/\/qoraapi.com\/blog\/detect-reduce-hallucinations\/\">detecting and reducing hallucinations<\/a>. Layer the judge on top for what no rule expresses: was the refusal appropriate, did the answer address the question, is the tone within policy.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">LLM judges have three biases you must design around<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Position bias.<\/strong> Comparing two answers, the judge prefers whichever came first, or last, more often than chance. Run each pair twice with the order swapped, count the comparison only when both runs agree, record disagreements as ties. That doubles judge cost and is worth it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Verbosity bias.<\/strong> Longer answers score higher even when the extra content is filler. Put an explicit length constraint in the rubric and, where length is irrelevant, state that longer is not better. Better still, measure length programmatically.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Self-preference.<\/strong> A model grading its own output scores it higher than a different model would. Use a judge from a different model family than the one under test, and do not silently swap the judge when you swap the model. Prefer pairwise comparison to absolute scoring for subjective criteria: &#8220;is A better than B&#8221; is more stable than &#8220;rate this one to seven&#8221;, and absolute scales drift between judge versions.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Writing a rubric a judge can apply consistently<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A rubric is code. If it is ambiguous it produces inconsistent verdicts, and that inconsistency surfaces as flakiness in your suite.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Define the boundary of pass explicitly.<\/strong> Not &#8220;the answer should be accurate&#8221;. Write: pass if every factual claim is supported by the provided context; fail if any claim is unsupported or contradicted. Then give one worked boundary case, such as a claim that is true but absent from the context, and state the verdict.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Use few discrete levels and name the middle one.<\/strong> A three-level scale of fail, partial and pass with an explicit definition of partial is easier to apply consistently than a one-to-ten scale. Define partial as a concrete condition: &#8220;supported and correct but omits one of the three required entities&#8221;. Treat partial as a fail for gating and report it separately: a partial that silently counts as 0.5 makes the pass rate hard to reason about. Require the judge to return structured JSON with a mandatory rationale, which is what lets a human audit it when a case flips.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Running evals in CI<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The core tension: a full suite with a judge costs money and minutes, and developers route around anything slow. Split the suite.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On every pull request, run 20 to 40 cases covering each tag bucket, scored deterministically plus a judge on the highest-risk tags only: under three minutes and a few cents. Nightly, run the full suite including the held-out slice, storing results with the model fingerprint and prompt hash attached.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Cache model responses.<\/strong> Key the cache on a hash of model identifier, prompt version hash, temperature, decoded input and fixture versions. If none changed, the recorded output is a valid replay, which turns most PR runs into pure scorer runs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Pin model versions.<\/strong> Never point the suite at a floating alias. Record the resolved model identifier in every result row and treat a change in it as a re-baseline event rather than a regression, because providers do update snapshots behind stable names. Routing every candidate through one OpenAI-compatible endpoint makes pinning and swapping tractable, which is one reason to keep eval traffic on a gateway such as <a href=\"https:\/\/qoraapi.com\/\">Qora<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">A single run cannot gate a merge<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Evals are stochastic: even at temperature zero, provider-side batching and hardware variation produce different outputs from identical inputs, so a one-case drop between two runs of the same commit is not evidence of anything. Gate on the deterministic scorers, which are stable: block the merge if a deterministic assertion fails on a case it previously passed, or if the pass rate falls by more than the interval width, and warn rather than block on smaller judge-based movement.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The statistics that actually matter<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How many cases before a difference is meaningful.<\/strong> The margin of error for a proportion is roughly one over the square root of n at 95 percent confidence, worst case near p = 0.5. So 100 cases give about plus or minus 10 points, 400 give about 5, and 1,600 give about 2.5. At 40 cases you are near plus or minus 15 points: a 40-case suite detects breakage and not much else, which is what pages you.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Report an interval, always.<\/strong> &#8220;89 percent&#8221; invites a comparison with &#8220;91 percent&#8221; that 100 cases cannot support. &#8220;89 percent, 95 percent CI [81, 94]&#8221; makes the comparison honest at a glance. For small n and rates near zero or one, use the Wilson interval rather than the normal approximation, which produces bounds below zero and behaves badly exactly where eval suites live.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Flakiness versus regression.<\/strong> Run the same code twice. A case that fails in both runs is a regression candidate. A case that fails once and passes once is flaky, either because behaviour is genuinely nondeterministic near a boundary or because the judge cannot apply the rubric consistently. Quarantine flaky cases with an owner rather than deleting them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What the harness itself costs<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Work the arithmetic before building: the judge is the line item that surprises people.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Take 100 cases, each with roughly 800 input tokens of prompt, fixture and question, producing roughly 300 output tokens. A full run is 80,000 input and 30,000 output tokens. Add a judge at 1,200 input tokens per case \u2014 rubric plus context plus answer \u2014 and 150 output tokens: another 120,000 input and 15,000 output. Call it 200,000 input and 45,000 output per full run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">At a blended $3 per million input tokens and $15 per million output tokens, that is $0.60 plus $0.68, so about $1.28 per run and roughly $39 a month nightly. Add a 40-case PR subset across 60 pull requests a month with a judge on half of them: 60 x 20 x 1,200 equals 1.44 million judge input tokens, about $4.32, plus a few dollars of output. The whole harness lands under $60 a month before caching, and caching pushes it far below, because a PR run only pays for cases whose inputs, fixtures or prompt actually changed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Latency follows the same shape. Judge calls take one to three seconds each, so a judge on every case dominates wall time; deterministic scorers first is what keeps a PR run short.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Drift: three things that change while you change nothing<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Eval suites fail in a specific way when unmaintained: they keep passing while the product gets worse.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>A provider silently updates a model.<\/strong> A snapshot alias keeps its name and changes its weights or serving stack. Record the resolved model identifier and, where the provider exposes one, a system fingerprint in every result row; a change in either is a re-baseline event, not a regression. Pinning to a dated snapshot reduces the frequency without eliminating it, since the same snapshot can be served on different hardware.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Your prompt file changes.<\/strong> Hash the fully rendered prompt after variable substitution, after tool definitions are serialised and after few-shot examples are included, not the template file on disk. Teams have shipped regressions because the hash covered the template while a fixture change altered the rendered text. Rollout practice is covered in <a href=\"https:\/\/qoraapi.com\/blog\/prompt-management-versioning\/\">prompt management and versioning<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Your retrieval index changes.<\/strong> A re-embedding job, a chunking tweak or a document deletion changes what the model sees without touching the prompt. Freeze retrieved context as a fixture so the suite tests generation deterministically, and run a separate retrieval eval measuring whether the expected document is recalled for each query: one eval cannot measure both, or you cannot attribute a failure. Traces carrying retrieved document IDs let you reconstruct which index state produced a bad answer, which is the job of <a href=\"https:\/\/qoraapi.com\/blog\/llm-observability\/\">LLM observability<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A result is only meaningful when you can name every input to the system \u2014 model identifier, prompt hash, fixture version, index version, scorer version. If any is missing from the row, the number is not comparable to next week&#8217;s.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A harness skeleton<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Four pieces: a case, a scorer protocol, a runner with a concurrency cap and a disk cache, and a report with intervals and a per-tag breakdown.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from __future__ import annotations\n\nimport asyncio\nimport hashlib\nimport json\nimport math\nimport re\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom typing import Any, Awaitable, Callable, Protocol, Sequence\n\n@dataclass(frozen=True)\nclass Case:\n    id: str\n    input: str\n    expect: dict[str, Any] = field(default_factory=dict)\n    context: list[str] = field(default_factory=list)\n    tags: tuple[str, ...] = ()\n    weight: float = 1.0\n    split: str = \"tune\"          # \"tune\" | \"holdout\"\n\n@dataclass(frozen=True)\nclass Verdict:\n    passed: bool\n    score: float                 # 0.0 .. 1.0, so partial credit stays visible\n    detail: str = \"\"\n\nclass Scorer(Protocol):\n    name: str\n    async def __call__(self, case: Case, output: str) -> Verdict: ...\n\nclass JsonSchemaScorer:\n    name = \"json_schema\"\n\n    def __init__(self, required: Sequence[str]) -> None:\n        self.required = required\n\n    async def __call__(self, case: Case, output: str) -> Verdict:\n        try:\n            payload = json.loads(output)\n        except json.JSONDecodeError as exc:\n            return Verdict(False, 0.0, \"not json: \" + exc.msg)\n        if not isinstance(payload, dict):\n            return Verdict(False, 0.0, \"top level is not an object\")\n        missing = [k for k in self.required if not payload.get(k)]\n        if missing:\n            return Verdict(False, 0.0, \"missing: \" + \",\".join(missing))\n        return Verdict(True, 1.0, \"\")\n\nclass GroundedNumbersScorer:\n    name = \"grounded_numbers\"\n    pattern = re.compile(r\"\\d+(?:[.,]\\d+)?\")\n\n    async def __call__(self, case: Case, output: str) -> Verdict:\n        haystack = \" \".join(case.context)\n        stray = [n for n in self.pattern.findall(output) if n not in haystack]\n        if stray:\n            return Verdict(False, 0.0, \"unsupported numbers: \" + str(stray[:5]))\n        return Verdict(True, 1.0, \"\")\n\ndef cache_key(model: str, prompt_hash: str, case: Case, seed: int) -> str:\n    blob = json.dumps(\n        {\n            \"model\": model,\n            \"prompt\": prompt_hash,\n            \"input\": case.input,\n            \"context\": case.context,\n            \"seed\": seed,\n        },\n        sort_keys=True,\n    )\n    return hashlib.sha256(blob.encode()).hexdigest()\n\nclass Cache:\n    def __init__(self, path: Path) -> None:\n        self.path = path\n        self.path.mkdir(parents=True, exist_ok=True)\n\n    def get(self, key: str) -> str | None:\n        f = self.path \/ (key + \".txt\")\n        return f.read_text(encoding=\"utf-8\") if f.exists() else None\n\n    def put(self, key: str, value: str) -> None:\n        (self.path \/ (key + \".txt\")).write_text(value, encoding=\"utf-8\")\n\n@dataclass\nclass Result:\n    case: Case\n    output: str\n    verdicts: list[Verdict]\n\n    @property\n    def passed(self) -> bool:\n        return all(v.passed for v in self.verdicts)\n\nasync def run_case(\n    case: Case,\n    generate: Callable[[Case], Awaitable[str]],\n    scorers: Sequence[Scorer],\n    cache: Cache,\n    model: str,\n    prompt_hash: str,\n    seed: int,\n) -> Result:\n    key = cache_key(model, prompt_hash, case, seed)\n    output = cache.get(key)\n    if output is None:\n        output = await generate(case)\n        cache.put(key, output)\n    verdicts = [await s(case, output) for s in scorers]\n    return Result(case, output, verdicts)\n\nasync def run_suite(\n    cases: Sequence[Case],\n    generate: Callable[[Case], Awaitable[str]],\n    scorers: Sequence[Scorer],\n    cache: Cache,\n    model: str,\n    prompt_hash: str,\n    concurrency: int = 12,\n    seed: int = 0,\n) -> list[Result]:\n    gate = asyncio.Semaphore(concurrency)\n\n    async def one(case: Case) -> Result:\n        async with gate:\n            return await run_case(\n                case, generate, scorers, cache, model, prompt_hash, seed\n            )\n\n    return await asyncio.gather(*(one(c) for c in cases))\n\ndef wilson(passed: int, n: int, z: float = 1.96) -> tuple[float, float]:\n    if n == 0:\n        return (0.0, 1.0)\n    p = passed \/ n\n    denom = 1 + z * z \/ n\n    centre = (p + z * z \/ (2 * n)) \/ denom\n    half = z * math.sqrt(p * (1 - p) \/ n + z * z \/ (4 * n * n)) \/ denom\n    return (max(0.0, centre - half), min(1.0, centre + half))\n\ndef report(results: Sequence[Result], split: str = \"tune\") -> dict[str, Any]:\n    rows = [r for r in results if r.case.split == split]\n    if not rows:\n        return {\"split\": split, \"n\": 0}\n\n    passed = sum(1 for r in rows if r.passed)\n    n = len(rows)\n    lo, hi = wilson(passed, n)\n\n    by_tag: dict[str, list[bool]] = {}\n    for r in rows:\n        for tag in r.case.tags:\n            by_tag.setdefault(tag, []).append(r.passed)\n\n    weight_total = sum(r.case.weight for r in rows)\n    weight_pass = sum(r.case.weight for r in rows if r.passed)\n\n    return {\n        \"split\": split,\n        \"n\": n,\n        \"pass_rate\": round(passed \/ n, 4),\n        \"ci95\": [round(lo, 4), round(hi, 4)],\n        \"weighted_pass_rate\": round(weight_pass \/ weight_total, 4),\n        \"by_tag\": {\n            tag: {\n                \"n\": len(v),\n                \"pass_rate\": round(sum(v) \/ len(v), 4),\n                \"ci95\": [round(x, 4) for x in wilson(sum(v), len(v))],\n            }\n            for tag, v in sorted(by_tag.items())\n        },\n    }\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two details are load-bearing. The cache key includes the prompt hash and the model identifier, so a prompt edit or model swap invalidates exactly the affected rows. And the report emits an interval next to every rate, because tag-level numbers come from the smallest samples.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A CI configuration<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A PR subset that blocks on deterministic failures, and a nightly full run across both splits.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>name: llm-evals\n\non:\n  pull_request:\n  schedule:\n    - cron: \"0 3 * * *\"        # nightly full run, 03:00 UTC\n\njobs:\n  pr-subset:\n    if: github.event_name == 'pull_request'\n    runs-on: ubuntu-latest\n    timeout-minutes: 10\n    steps:\n      - uses: actions\/checkout@v4\n      - uses: actions\/setup-python@v5\n        with:\n          python-version: \"3.12\"\n          cache: pip\n      - run: pip install -r evals\/requirements.txt\n      - name: Run PR subset\n        env:\n          QORA_API_KEY: ${{ secrets.QORA_API_KEY }}\n        run: |\n          python -m evals.run \\\n            --suite suites\/support-agent.yaml \\\n            --split tune \\\n            --subset-per-tag 4 \\\n            --scorers json_schema,grounded_numbers,citation_exists \\\n            --judge-tags risk:high \\\n            --judge-samples 1 \\\n            --cache .evalcache \\\n            --report pr-report.json \\\n            --fail-on-deterministic \\\n            --max-pass-rate-drop 0.05\n      - uses: actions\/upload-artifact@v4\n        if: always()\n        with:\n          name: pr-report\n          path: pr-report.json\n\n  nightly-full:\n    if: github.event_name == 'schedule'\n    runs-on: ubuntu-latest\n    timeout-minutes: 45\n    strategy:\n      matrix:\n        split: [tune, holdout]\n    steps:\n      - uses: actions\/checkout@v4\n      - uses: actions\/setup-python@v5\n        with:\n          python-version: \"3.12\"\n          cache: pip\n      - run: pip install -r evals\/requirements.txt\n      - name: Run full suite\n        env:\n          QORA_API_KEY: ${{ secrets.QORA_API_KEY }}\n        run: |\n          python -m evals.run \\\n            --suite suites\/support-agent.yaml \\\n            --split ${{ matrix.split }} \\\n            --scorers json_schema,grounded_numbers,citation_exists,judge \\\n            --judge-model claude-sonnet-4-5 \\\n            --judge-samples 3 \\\n            --runs 3 \\\n            --concurrency 16 \\\n            --cache .evalcache \\\n            --record-model-fingerprint \\\n            --baseline baselines\/${{ matrix.split }}.json \\\n            --report full-${{ matrix.split }}.json\n      - uses: actions\/upload-artifact@v4\n        with:\n          name: full-report-${{ matrix.split }}\n          path: full-${{ matrix.split }}.json\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>--runs 3<\/code> flag separates flakiness from regression: mark a case failed only if it fails in every run. The <code>--baseline<\/code> flag compares against the last accepted full run rather than an absolute threshold, which is the only comparison that survives normal drift.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What to build in week one<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Week one:<\/strong> forty cases from the last two weeks of logs and the last ten incidents, stored as YAML in the repository. Deterministic scorers only: schema, required fields, citation existence, number grounding, length. A runner of about a hundred lines that writes a JSON report with Wilson intervals, plus disk caching. A CI job on every PR that blocks on deterministic failures only.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Weeks two to four:<\/strong> add the judge with a written rubric for the two or three criteria you cannot express mechanically, and calibrate it against thirty human-labelled outputs before you trust it. Add the nightly full run and the held-out split, and start recording model fingerprints.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>What can wait:<\/strong> a web dashboard, a results database, an annotation UI, pairwise model comparison, automatic prompt optimisation, multi-turn evals, and fine-tuning the judge. None find a regression in week one. The most common failure mode is building the framework instead of the cases: a hundred hand-written cases with a sixty-line runner beats an elegant plugin architecture holding eight placeholders.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">How is an eval harness different from unit testing?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Unit tests assert on deterministic functions; an eval harness asserts on a distribution. A single failure may be sampling noise rather than a bug, so you report rates with intervals instead of a binary build status, and the expected value is usually a set of properties. Keep ordinary unit tests for the parsing, routing and prompt-rendering code around the model: those are exact, fast, and catch regressions that never needed a model call.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should I use an off-the-shelf eval framework?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Use one if it fits, but keep the case format and the scorers yours. The case file and scorer implementations encode what your product must do; the runner, caching and reporting are commodity. Teams that adopt a framework&#8217;s case format wholesale often find it cannot express their assertions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How often should I update the golden set?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Add cases continuously from incidents: every user-visible failure gets one. Prune rarely, and only when a case duplicates another and covers no distinct tag combination. Never delete a case because it fails, and review the set quarterly for cases that assert an implementation detail rather than a product property.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Do I need a judge if my outputs are structured?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. If every output is a JSON object validated against a strict schema and every field is checkable against the input, deterministic scorers cover you. Judges earn their place when the output is prose, when the criterion is subjective such as tone or the appropriateness of a refusal, or when correctness needs reasoning over the context.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How do I evaluate a multi-turn agent?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Make the trajectory the unit of assertion, not the final message. Record tool calls with their arguments and assert on the sequence: no write before a confirmation, no more than N tool calls, the final answer cites a tool result. For the judge, feed the whole trajectory and the tool results, not just the last message.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">An application eval harness is a small amount of machinery around a hard part: deciding what &#8220;still works&#8221; means for your feature, in a form a machine can check. Cases come from production, assertions are properties rather than strings, scorers run cheapest first, and results are reported with confidence intervals rather than a single number.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Two disciplines hold it together. Pin and record everything that feeds the system, so a result is comparable to the previous one. And treat the harness as production code with an owner, because an unmaintained suite drifts into passing while the product regresses, which is worse than having no suite at all.<\/p>\n\n\n\n\n<h3 class=\"wp-block-heading\">Related reading<\/h3>\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/qoraapi.com\/blog\/evaluate-benchmark-ai-models\/\">Evaluating and Benchmarking AI Models Before You Ship<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/prompt-management-versioning\/\">Prompt Management and Versioning in Production<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/choose-right-ai-model-routing\/\">How to Choose the Right AI Model: A Practical Model-Routing Guide<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ab-testing-prompts-models\/\">A\/B Testing Prompts and Models in Production<\/a><\/li><\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>Public benchmarks do not predict your app. Build a task-specific eval suite: golden cases from real traffic, the right scorer per assertion, LLM-judge bias controls, and CI gating that survives stochastic output.<\/p>\n","protected":false},"author":1,"featured_media":299,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,6,9,7],"class_list":["post-300","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-api","tag-ai-api","tag-api-gateway","tag-developer-tools","tag-developers"],"_links":{"self":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/300","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/comments?post=300"}],"version-history":[{"count":1,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/300\/revisions"}],"predecessor-version":[{"id":310,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/300\/revisions\/310"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media\/299"}],"wp:attachment":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media?parent=300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/categories?post=300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/tags?post=300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}