A hallucination is a grounding failure, not a random error: the model asserted something your evidence set does not support. Reducing it is an engineering loop — define the evidence set, verify each atomic claim against it, refuse when unsupported, and track groundedness rate on a labelled eval set that includes unanswerable questions.
What hallucination actually is (and why each mode needs its own fix)
Define the evidence set E for a request: the retrieved chunks, tool results, or database rows you actually handed the model. An answer is hallucinated to the degree it asserts claims E does not entail. Same model, same prompt, two retrievers — two hallucination rates. Hallucination is a rate you bound and monitor, not a bug you close.
| Failure mode | What went wrong | Symptom to look for | Primary fix |
|---|---|---|---|
| Grounding failure | Right evidence in context, but the model answered from parametric memory or blended two sources | Claim exceeds the cited chunk; citation resolves to a chunk that does not contain it | Claim-level groundedness check, citation verification, refusal |
| Knowledge gap / retrieval miss | The evidence was never retrieved, or is absent from the corpus | Answer with no citations, or citations to adjacent-but-wrong chunks | Measure retrieval recall; rewrite queries; expect refusals |
| Instruction drift | Evidence present, but scope or format constraints were ignored | Over-answering: fills unrequested fields, invents enum values | Constrained decoding, a written contract, boundary validation |
| Entity / numeric conflation | Correct evidence, wrong binding — attribute of A attached to B | Right tokens, wrong pairing; passes a whole-answer judge | Claim-level verification with a required verbatim span |
Instrument by mode. The largest row tells you where the next sprint goes — usually retrieval, not prompting.
Detection: four layers, cheapest first
Run detection as an escalating ladder and short-circuit: if a cheap layer proves the answer is ungrounded, you never pay for the expensive one. Ordering matters more than the judge model, because it sets your cost per answer. Layers 1 and 2 run on 100% of user-visible answers, layer 3 is a 1–5% sample, and layer 4 belongs offline.
Layer 1: deterministic citation verification
Parse citations from the answer and assert every id exists in the retrieved set. This is free and catches the most common production failure: the model cites [3] when only two chunks were retrieved. Run it as a gate, not a metric — once citation existence fails, the answer is known-bad and no judge call is needed.
Layer 2: a working groundedness check
Decompose the answer into atomic claims — one fact each — and grade each claim against the chunk it cites, never against the whole context. Whole-answer judging hides conflation: an answer that misattributes one attribute out of five scores as “mostly correct”. Three details make the check trustworthy: require a verbatim span, reject spans that are not substrings of the cited chunk (that catches judge hallucination), and treat partial as unsupported.
import json, re
from openai import OpenAI
client = OpenAI(base_url="https://api.example.com/v1", api_key="...")
CLAIM_SCHEMA = {"type": "json_schema", "json_schema": {
"name": "claims", "strict": True, "schema": {
"type": "object",
"properties": {"claims": {"type": "array", "items": {
"type": "object",
"properties": {
"claim": {"type": "string"},
"chunk_id": {"type": "string"},
"verdict": {"type": "string",
"enum": ["supported", "partial", "unsupported"]},
"span": {"type": "string"}, # verbatim from the chunk
},
"required": ["claim", "chunk_id", "verdict", "span"],
"additionalProperties": False}}},
"required": ["claims"], "additionalProperties": False}}}
JUDGE_PROMPT = """Split the ANSWER into atomic claims (one fact each) and grade
each claim against the chunk it cites, using ONLY that chunk.
- chunk_id must be one of the answer's citations.
- span must be copied VERBATIM from that chunk, or "" if nothing supports it.
- "partial" means only part of the claim is supported; treat it as unsupported.
CHUNKS:
{chunks}
ANSWER:
{answer}
"""
def check_groundedness(answer, chunks, judge_model="judge-model-id"):
ids = {c["id"] for c in chunks}
# Layer 1, free: does every cited id exist in the evidence set?
fabricated = sorted(set(re.findall(r"\[([A-Za-z0-9_.\-]+)\]", answer)) - ids)
block = "\n\n".join(f'[{c["id"]}] {c["text"]}' for c in chunks)
resp = client.chat.completions.create(
model=judge_model, temperature=0, response_format=CLAIM_SCHEMA,
messages=[{"role": "user",
"content": JUDGE_PROMPT.format(chunks=block, answer=answer)}])
claims = json.loads(resp.choices[0].message.content)["claims"]
text_by_id = {c["id"]: c["text"] for c in chunks}
for c in claims:
# Reject a hallucinating judge: the span must exist in the cited chunk.
if c["verdict"] == "supported" and c["span"].strip() not in text_by_id.get(c["chunk_id"], ""):
c["verdict"] = "unsupported"
supported = sum(1 for c in claims if c["verdict"] == "supported")
return {"groundedness": supported / len(claims) if claims else 0.0,
"unsupported": [c["claim"] for c in claims if c["verdict"] != "supported"],
"fabricated_citations": fabricated}
Two cost controls: batch every claim of one answer into a single judge call, and route the judge to a small model when claims are short and extractive.
Layer 3: self-consistency sampling, with its blind spot
Sample the same question k times (k of 3–5) at non-zero temperature, extract atomic claims from each sample, normalise and cluster them. A claim appearing in fewer than half the samples is low-confidence. Cost scales linearly in k, so use this on high-value answers, not every request.
The blind spot is decisive: self-consistency detects variance, not error. A model that consistently misreads the same chunk produces five identical wrong answers and passes cleanly.
Layer 4: LLM-as-judge with a rubric you can defend
Score on anchored levels, not vibes: 5 — every claim entailed by its cited evidence; 3 — main claim supported, at least one detail is not; 1 — core claim unsupported or contradicted. Three rules make the score usable:
- Use a different model family for the judge than for the generator. Self-preference bias inflates scores precisely on the answers you worry about.
- Force quoted spans — a judge that cannot quote its evidence is guessing — and calibrate it against 50–100 hand-labelled answers before you trust the number.
- Pin the judge version. An upgrade changes your historical metric series overnight.
Retrieval grounding: mandatory citations, valid refusals
- Cite-then-write, enforced in code. Require the chunk id before the claim it supports, then delete or refuse any sentence with no verified citation. Fabrication becomes visible in the token stream, and asking the model to self-censor is not a control.
- Refusal is a first-class output. Return an explicit status such as
insufficient_evidence, then track it both ways: refusal correctness on unanswerable questions and over-refusal rate on answerable ones. A refusal rate of zero means the policy does not exist. - Retrieval recall is the ceiling. Groundedness cannot exceed recall — if the correct chunk is not in the top-k, no prompt recovers it. Measure recall@k against gold chunk labels and fix retrieval first. For chunking, hybrid search, and reranking, see production RAG.
- Watch the retrieval score margin. Top-1 minus top-2 similarity is available before you spend an output token, and a thin margin means near-duplicate or conflicting chunks — exactly where conflation happens.
Structured outputs: constrain the shape, not the truth
Schema-constrained decoding removes a whole class of failures — malformed JSON, invented enum members, missing fields, prose where a value belongs. It does not make facts true. Schema validity is a format guarantee; grounding is a separate, verified property. Used deliberately, schemas still cut fabrication three ways:
- Make refusal representable. An
insufficient_evidenceboolean plus an explicitunknownenum member gives the model a legal way out. If the only legal output is a filled-in answer, you have instructed it to fabricate. - Order fields as a decision. Put
insufficient_evidencebeforeanswer. Decoders emit in order, so the model commits to whether the evidence suffices before writing a fluent answer it would otherwise rationalise. - Cite per claim, not per answer. Ask for
citations: [{"claim": ..., "chunk_id": ...}]instead of a flat list, which lets the model attach the right document to the wrong sentence and still look sourced.
For schema mechanics — strict mode, nested objects, nullable fields, provider differences — see structured outputs. To test whether a schema or a better model buys you more, our guide on evaluating AI models covers measuring it instead of guessing.
Guardrails and fallback: thresholds from the cost of being wrong
Decide the response policy before you ship, in a config file rather than a prompt. A workable three-tier default:
| Signal state | Response | Why |
|---|---|---|
| Groundedness high, citations verified, wide retrieval margin | Answer normally | Evidence is unambiguous; hedging only adds noise |
| Groundedness mid, or one claim unverified | Answer the grounded part, show sources, mark the rest unverified | Partial value beats a full refusal if the gap is visible |
| Groundedness low, any fabricated citation, thin margin | Refuse with insufficient_evidence, offer escalation | Once evidence does not support the answer, fluent output is a liability |
Set thresholds from the cost of a wrong answer, not the average score. Regulated advice warrants refusing below a high bar; internal search can show a hedged answer with sources because the user can check it. Combine signals with hard ANDs for safety-critical fields: one fabricated citation should force a refusal however high the judge score is, since a weighted average would let a good score mask a fabricated source.
The fallback ladder, in order: re-retrieve with a rewritten query; re-answer with a stronger model on the same evidence; narrow the answer to the grounded portion; refuse and escalate to a human queue. The one move never on the ladder is falling back to ungrounded generation. Log every escalation with its reason and promote those cases into your eval set — they are your best labelled data, because the system has already flagged them.
An evaluation harness that measures groundedness, not vibes
Build 150–300 eval items, each with a question, gold chunk ids (or an unanswerable flag), and a reference answer. Composition matters more than size: roughly 60% answerable, 25% unanswerable, and 15% conflicting or adversarial, where two chunks disagree and the correct behaviour is to surface the conflict rather than pick silently.
The unanswerable class is mandatory: on an answerable-only set, a model that never refuses scores perfectly, so you would be measuring fluency and calling it groundedness. Report five metrics per prompt_version — groundedness rate (supported claims / total claims); citation precision (claims whose cited chunk supports them / claims carrying a citation); refusal correctness (refused unanswerable / all unanswerable); over-refusal rate (refused answerable / all answerable); hallucination rate (answers with an unsupported claim or fabricated citation / all answers). Pair over-refusal with refusal correctness, or you will “fix” hallucination by refusing everything.
def run_eval(items, answer_fn):
"""items: [{"question":..., "answerable":bool}]
answer_fn: your app -> {"status":..., "answer":..., "chunks":[...]}"""
m = dict(grounded=0, hallucinated=0, refused_ok=0, refused_wrong=0)
for it in items:
out = answer_fn(it["question"])
refused = out["status"] == "insufficient_evidence"
if not it["answerable"]:
# Answering an unanswerable question is a hallucination by definition.
m["refused_ok" if refused else "hallucinated"] += 1
continue
if refused:
m["refused_wrong"] += 1
continue
v = check_groundedness(out["answer"], out["chunks"])
clean = v["groundedness"] == 1.0 and not v["fabricated_citations"]
m["grounded" if clean else "hallucinated"] += 1
n_ans = sum(1 for i in items if i["answerable"])
n_un = len(items) - n_ans
return {"groundedness_rate": m["grounded"] / max(n_ans, 1),
"refusal_correctness": m["refused_ok"] / max(n_un, 1),
"over_refusal_rate": m["refused_wrong"] / max(n_ans, 1),
"hallucination_rate": m["hallucinated"] / len(items)}
Then gate CI on it: no change to prompt, retriever, chunker, or model may drop groundedness rate by more than a couple of points or push over-refusal outside its band. Without a gate, prompt edits ship on three hand-picked examples and regressions surface weeks later as vague user complaints.
Observability: log the evidence, not just the answer
Emit one structured event per request: request_id, prompt_version, model id, retrieved chunk ids with similarity scores, the raw answer, parsed citations, per-claim verdicts, judge model and version, decision tier, escalation reason, latency, and token counts. Three details separate a real audit trail from a decorative one:
- Log a content hash per chunk, not just its id. Re-indexing changes chunk text under a stable id, so without a hash your audit trail points at text the model never saw.
- Sample the expensive judge smartly. Log cheap fields on all traffic; run groundedness checks on 1–5% of ordinary answers plus 100% of refusals, escalations, and low-tier decisions.
- Store the retrieved context for replay. Regressions are only debuggable if you can re-run a request offline against a new prompt with the exact original evidence set.
Dashboard groundedness rate and citation precision by prompt_version, and alert on a sustained drop rather than a single bad answer. For the wider logging stack — traces, spans, cost attribution — see our LLM observability guide. One note on judge diversity: keeping generator and judge on different model families is a config change rather than a rewrite when both sit behind one OpenAI-compatible endpoint, which is what an AI API relay such as qoraapi.com provides.
Frequently asked questions
Can hallucinations be eliminated entirely?
No. A generative model can always produce an unsupported claim, so the goal is bounding and measuring. Drive the visible rate down with retrieval grounding, claim-level verification, refusal paths, and deterministic code for anything computable, then keep a human escalation route for the residue.
Does setting temperature to zero stop hallucinations?
No. Temperature controls variance, not grounding. Lowering it makes fabrication more consistent — easier to catch, no less wrong — and removes self-consistency sampling as a signal, since every sample becomes identical. Missing evidence still yields a confident parametric answer.
Do larger models hallucinate less?
Frontier models are better at honouring evidence constraints and abstaining when instructed, but they fabricate more fluently, so failures are harder to spot. Retrieval quality, mandatory citations, and a real refusal policy usually move groundedness further than a tier upgrade.
Should detection run per request or only in evaluation?
Split by cost. Deterministic citation checks and a small-model claim check can run on every user-visible answer; self-consistency sampling and frontier judges belong in a 1–5% production sample plus your offline eval. Any check whose failure changes what the user sees must run inline as a gate.
Conclusion
Hallucination reduction is not a prompt trick, it is a pipeline: an evidence set you control, atomic claims verified against it, mandatory citations, a schema that makes refusal legal, guardrails whose thresholds come from the cost of being wrong, and an eval harness containing the unanswerable questions most teams never write.
Start where the leverage is: fix retrieval recall, then add the groundedness check above as both a runtime gate and an offline metric. If you want generator and judge on different model families through one endpoint, integrating an AI API takes an afternoon.
