Building Production RAG: Chunking, Hybrid Search, and Re-Ranking

Production RAG fails in three measurable places: chunks that split an answer across a boundary, pure vector search that misses exact identifiers, and no re-ranking, so a mediocre retriever feeds noise into a good model. Fix retrieval in that order — semantic chunking, BM25 plus vector fusion, cross-encoder re-ranking — before you touch the prompt.

This is the follow-up to embeddings and RAG, which covers what embeddings are and how to call them. This article assumes you already have a working vector index and your answers are still wrong, and covers the four retrieval stages that close the gap.

Why naive RAG fails in production

A demo works because you wrote the five test questions yourself. Production breaks in four diagnosable ways.

• Chunk boundaries cut answers in half. The retriever returns the chunk containing the question but not the chunk containing the answer. This is the most common cause of “the answer is in the docs but the bot says it isn’t” — and it stays invisible unless you inspect retrieved chunks, not just the final text.
• Dense embeddings are lexically blind. A bi-encoder compresses rare tokens into a general region of the vector space, so ERR_CONN_REFUSED_0x5, SKU A-4471-B, and invoice.total_cents all land near semantically similar but wrong neighbours. Systems that “work for questions and fail for lookups” are almost always failing here.
• Top-k dilutes signal. Bi-encoder similarity is not calibrated relevance: rank 1 means “least dissimilar”, not “correct”. Passing five chunks when one matters gives the model four opportunities to anchor on the wrong context, and content buried in the middle of a long context is used less reliably.
• Nobody measured retrieval. Teams rewrite prompts for weeks while recall@10 sits at 0.5. The generator cannot fix a document that was never retrieved.

The operating rule: if recall@10 is below roughly 0.85, stop tuning the prompt. Prompt engineering cannot recover an answer that never entered the context window. Everything below is retrieval work.

Chunking strategies: set the size from your data, not from a blog post

Two measurements decide chunk size for you. First, the answer span: the median character length of the passage a correct answer actually needs. FAQ corpora need 200–400 characters; API references with multi-step procedures need 800–1500. Second, query specificity: exact-lookup queries want smaller, more precise chunks, while “explain the architecture” queries want larger ones. Sample 50 real queries, label the minimum passage that answers each, and take the 75th percentile of that length. That number is your target size.

Fixed-size splitting with overlap is predictable and cheap, but it cuts mid-thought. Recursive or structural splitting — headings, then paragraphs, then sentences — is the correct default. Semantic chunking, where you embed sentences and cut where consecutive-sentence similarity drops below a threshold, sounds better than it usually is: it produces wildly variable chunk sizes, which breaks your embedding cost model and can emit 40-token fragments that carry no retrievable signal. If you use it, clamp it to a band (say 200–1200 characters) and fall back to recursive splitting inside those bounds.

Overlap deserves more scepticism than it gets. Overlap does not add information; it duplicates it, inflating storage and embedding cost and producing near-duplicate hits that crowd out diversity in your top-k. Apply overlap only at paragraph seams, never inside tables or code, and deduplicate by content hash at retrieval time.

Content type	Splitter	Size / overlap	Why
Prose docs, articles	Recursive on headings then paragraphs	600–1000 chars, 10–15% overlap	Preserves argument flow; overlap only at paragraph seams
API reference, config	Structural + heading breadcrumb	300–700 chars, no overlap	Each endpoint is self-contained; the breadcrumb restores the context the size removes
Tables, spec matrices	Atomic, rows flattened to key:value	One table per chunk set, no overlap	A data row without its header row is unrecoverable
Source code	AST: function and class boundaries	One symbol per chunk, signature prepended	The retrieval target is a symbol, not a byte range
FAQ, support tickets	One Q&A pair per chunk	150–400 chars, no overlap	Matches the shape of the incoming query distribution
Contracts, policies	Clause-level, by numbering	Clause boundaries, no overlap	Citations must map back to a clause number a human can verify

Two structural moves pay for themselves immediately. Never split a table — and flatten each row into column: value lines so column names become lexically searchable, which is exactly what the hybrid stage below needs. Prepend the heading path to every chunk: a chunk reading “Set the timeout to 30” is nearly useless, while “Payments API > Retries > Configuration — Set the timeout to 30” is retrievable by three different query phrasings. Both are cheap to implement and both are pure retrieval-quality gains.

import re

def split_sections(md: str):
    """Split markdown on headings, keeping code fences and tables atomic."""
    parts, cur, path, in_fence = [], [], [], False
    for line in md.splitlines():
        if line.startswith("```"):
            in_fence = not in_fence
        if not in_fence and re.match(r"^#{1,6}\s", line):
            if cur:
                parts.append(("\n".join(path), "\n".join(cur).strip()))
            level = len(line) - len(line.lstrip("#"))
            path = path[: level - 1] + [line.lstrip("# ").strip()]
            cur = [line]
        else:
            cur.append(line)
    if cur:
        parts.append(("\n".join(path), "\n".join(cur).strip()))

    # Breadcrumb every chunk so short splits stay retrievable.
    return [{"id": str(i), "breadcrumb": head, "text": f"{head}\n{body}"}
            for i, (head, body) in enumerate(parts) if body]


def flatten_table(header: str, rows: list) -> str:
    """Turn a markdown table into key:value lines so BM25 can hit column names."""
    cols = [c.strip() for c in header.strip("|").split("|")]
    out = []
    for row in rows:
        cells = [c.strip() for c in row.strip("|").split("|")]
        out.append("; ".join(f"{c}: {v}" for c, v in zip(cols, cells) if v))
    return "\n".join(out)

Hybrid search: BM25 plus vectors, fused with reciprocal rank fusion

BM25 scores exact term overlap with inverse document frequency weighting, which makes it unbeatable on rare tokens: error codes, function names, part numbers, version strings. Dense retrieval handles paraphrase and synonymy, where BM25 scores zero because the words differ. They fail in opposite directions, so the union is strictly better than either — and hybrid retrieval is the highest-value single change most RAG systems can make.

The fusion problem is that the two score scales are incomparable. BM25 scores are unbounded and depend on corpus statistics; cosine similarities are bounded and depend on the embedding model. Min-max normalising them per query looks reasonable and is unstable in practice, because the normalisation is driven by whatever happened to be in that query’s result set.

Reciprocal Rank Fusion (RRF) avoids the problem entirely by discarding the scores and fusing the ranks: score(d) = Σ wr / (k + rankr(d)). The constant k, usually 60, damps the influence of the very top ranks; smaller values make each retriever’s top-1 dominate, larger values flatten contributions across the list. Sixty is a robust default and rarely worth tuning before you have an eval set — tune the per-retriever weights wr first, since that is where real bias lives.

One detail decides whether RRF works at all: how deep you retrieve from each side. Fuse the top 50 from each retriever, not the top 10. A document ranked 30th by both retrievers is a strong relevance signal that never enters a shallow pool.

import numpy as np
from collections import defaultdict


def rrf_fuse(rankings, k=60, weights=None):
    """rankings: list of ranked id lists (best first). Returns id -> fused score."""
    weights = weights or [1.0] * len(rankings)
    fused = defaultdict(float)
    for ranking, w in zip(rankings, weights):
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] += w / (k + rank)
    return dict(sorted(fused.items(), key=lambda kv: -kv[1]))


def hybrid_search(query, chunks, bm25, faiss_index, embed,
                  depth=50, w_lex=1.0, w_dense=1.0):
    """BM25 + dense retrieval, fused by reciprocal rank fusion."""
    ids = [c["id"] for c in chunks]

    # 1. lexical side - exact terms, rare identifiers, column names
    lex_scores = bm25.get_scores(query.lower().split())
    lexical = [ids[i] for i in np.argsort(-lex_scores)[:depth]]

    # 2. dense side - paraphrase and synonymy
    qv = np.asarray([embed(query)], dtype="float32")
    _, idx = faiss_index.search(qv, depth)
    dense = [ids[i] for i in idx[0] if i != -1]

    # 3. rank-based fusion - no score normalisation needed
    fused = rrf_fuse([lexical, dense], k=60, weights=[w_lex, w_dense])
    return list(fused)[:depth]

Use w_lex=2.0 when your corpus is identifier-heavy (logs, code, SKUs, legal citations) and w_dense=2.0 when queries are conversational and users rarely type exact terms.

Re-ranking: take top-K from hybrid, return top-N to the model

A bi-encoder encodes the query and the document independently, so it can never model the interaction between their terms. A cross-encoder concatenates query and document and scores them jointly, which is substantially more accurate — and costs one forward pass per candidate. That cost structure is exactly why it belongs in a second stage: you cannot run it over a corpus, but you can run it over 50 candidates.

def retrieve_and_rerank(query, chunks, bm25, faiss_index, embed, rerank,
                        k_retrieve=50, k_final=6):
    """Stage 1: hybrid recall@50. Stage 2: cross-encoder precision@6."""
    candidates = hybrid_search(query, chunks, bm25, faiss_index, embed,
                               depth=k_retrieve)
    by_id = {c["id"]: c for c in chunks}
    docs = [by_id[cid]["text"] for cid in candidates]

    # One batched call - per-document HTTP overhead dominates at K=50.
    result = rerank(query=query, documents=docs, top_n=k_final)

    # Map reranker positions back to the original chunk objects.
    return [by_id[candidates[r["index"]]] for r in result["results"]]

Three things matter more than the choice of reranker:

• Re-ranking changes your optimal chunk size. With a reranker in the pipeline you can retrieve small, precise chunks — better lexical match, less noise in the vector — and then expand to the parent chunk before generation. This “small-to-big” pattern is often a larger quality win than the reranker itself, and it only becomes safe once a cross-encoder is filtering the pool.
• Cap K at the knee. Cross-encoder latency grows roughly linearly in the number of candidates. K=50 is usually the knee; pushing to 200 buys a point or two of recall for several times the rerank latency.
• Watch for the flip. If reranking consistently demotes your top BM25 hit, your lexical weight is too high — you are promoting chunks that match surface terms without answering the question.

Query rewriting and metadata filtering

Hypothetical document embeddings (HyDE). Instead of embedding the user’s query, ask a small model to write a short hypothetical answer and embed that. The intuition is sound: an answer-shaped passage sits closer in embedding space to real answer chunks than a six-word question does. The routing rule matters more than the technique — enable HyDE when the query is under about five tokens or is an open “how/why” question, and disable it for exact-lookup queries, where a fabricated hypothetical answer actively pulls the query vector away from the correct chunk. Because it adds a full generation call, cache the hypotheticals: query traffic is heavily skewed.

Multi-query expansion — three paraphrases, retrieve for each, fuse with RRF — improves recall and triples retrieval cost. Use it on the recall-critical path only.

Metadata filtering is where production systems break quietly. Four rules:

• Pre-filter, do not post-filter. Dropping chunks after ranking fails badly on selective filters: if a tenant is 1% of the corpus, a 50-document candidate pool contains roughly half of one of their documents. Apply filters inside the ANN search, or over-fetch by at least 1/selectivity.
• Design the facets you will actually filter on: tenant or workspace id, document type, source system, effective date or version, and access level. Enforce them at the index layer, never by asking the model to ignore content.
• Version and date filters are the cheapest fix for stale answers. “Latest policy” without a date filter will surface a superseded document that is semantically identical to the current one — the model has no way to prefer the newer text.
• Access control is a hard filter, not a ranking signal. A post-filter that removes unauthorised chunks after ranking still leaks their existence through scores and ordering, and can be defeated by increasing k. Retrieval is a security boundary.

Evaluation: does retrieval actually help?

Evaluate the two layers separately, or you will never know which one to fix. Retrieval quality is measured against a labelled set of query-to-relevant-chunk mappings: recall@k tells you whether the answer is even in the pool, which is the ceiling on end-to-end accuracy, while MRR and nDCG@10 tell you whether it is near the top, which is what re-ranking and context ordering control. Answer quality is measured on the generated text: groundedness (does every claim map to a cited chunk?), citation precision, and abstention correctness.

The non-obvious requirement: include unanswerable queries. Most RAG eval sets contain only questions the corpus can answer, so they cannot detect the failure mode that destroys user trust fastest — a confident answer assembled from irrelevant context. Aim for roughly 10% unanswerable queries and track the leak rate.

Metric	Stage	What it catches	Practical target
recall@50	Retrieval (hybrid)	Answer absent from the candidate pool — the accuracy ceiling	> 0.90
recall@5	Post-rerank	Whether re-ranking actually improved ordering	> 0.75
MRR / nDCG@10	Ranking	Relevant chunk buried under near-duplicate noise	MRR > 0.70
Citation precision	Generation	Model citing chunks that do not support the claim	> 0.90
Abstention leak rate	End to end	Confident answers built from irrelevant context	< 5%
p95 latency	System	Whether the extra stages are actually shippable	Defined by your budget

An offline harness is a hundred lines. Label at chunk level, not document level — document-level labels hide chunking failures, which is precisely what you are trying to detect.

def evaluate(retrieve, dataset, ks=(1, 5, 10, 50)):
    """dataset: [{"query": str, "relevant": set[chunk_id], "answerable": bool}]"""
    hits = {k: 0 for k in ks}
    rr_sum, leaks = 0.0, 0
    answerable = [row for row in dataset if row["answerable"]]
    unanswerable = [row for row in dataset if not row["answerable"]]

    for row in answerable:
        ranked = [c["id"] for c in retrieve(row["query"])]
        first = next((i for i, cid in enumerate(ranked, 1)
                      if cid in row["relevant"]), None)
        rr_sum += 1.0 / first if first else 0.0
        for k in ks:
            hits[k] += bool(set(ranked[:k]) & row["relevant"])

    for row in unanswerable:
        ranked = [c["id"] for c in retrieve(row["query"])]
        leaks += bool(ranked and ranked[0] not in row["relevant"])

    n = len(answerable)
    report = {f"recall@{k}": round(hits[k] / n, 3) for k in ks}
    report["mrr"] = round(rr_sum / n, 3)
    report["leak_rate"] = round(leaks / max(len(unanswerable), 1), 3)
    return report


# Ablation gate: run before and after every index or prompt change.
# baseline             recall@5 0.61  mrr 0.58
# + semantic chunking  recall@5 0.72  mrr 0.66
# + hybrid + rrf       recall@5 0.79  mrr 0.71
# + cross-encoder      recall@5 0.86  mrr 0.83

Build the labelled set from real query logs plus every production failure, verbatim. 150–300 queries is enough to detect meaningful regressions if the set spans query types — lookups, comparisons, multi-hop questions — because coverage beats raw size. Treat recall@10 as a regression gate in CI. For the answer-quality layer, the same discipline that makes model outputs gradeable applies: fixed schemas and deterministic scoring, as covered in our guide to structured outputs.

Cost and latency of the extra stages

Latency in a RAG pipeline is additive. For an interactive Q&A path the shape is stable: query embedding is one API round trip; BM25 runs in-process in single-digit milliseconds; ANN search over 10⁵–10⁶ vectors is a few tens of milliseconds; RRF fusion is arithmetic; cross-encoder re-ranking over K=50 is the dominant added stage; generation is usually the largest component of all.

Costs follow the same asymmetry. Re-ranking is priced per document, so it scales with K times query volume — but you are scoring a few dozen short passages, which typically makes it a small fraction of the token cost of generating the answer. HyDE, by contrast, adds an entire generation call, the same order of magnitude as the answer itself. Re-ranking is cheap precision; rewriting is expensive recall. That one distinction explains most of the design decisions below.

• Add hybrid search unconditionally. BM25 is in-process, RRF is arithmetic, and the only added cost is a slightly larger candidate pool.
• Add a reranker when recall@50 is much higher than recall@5. That gap is precisely the precision the cross-encoder recovers. If recall@5 already equals recall@50, your retriever is already precise and re-ranking buys latency for nothing.
• Add HyDE only if a meaningful share of traffic is short and vague. Route it by query length and intent instead of applying it globally, and cache the generated hypotheticals.
• Add multi-query expansion last, and only where recall is critical. It is the only stage that multiplies retrieval cost by design.

The sequencing matters as much as the stages. Fix chunking first — it is free and delivers the largest single jump. Then add BM25 and RRF, which are nearly free. Then re-ranking, which costs a little. Rewriting comes last. Teams that start with query rewriting pay the most and improve the least, because they are rewriting queries against a corpus that was chunked badly. Serving embeddings and reranking through one OpenAI-compatible endpoint keeps this from turning into a vendor-management problem; qoraapi.com exposes embedding and rerank models behind a single API key, so the pipeline above stays one credential and one retry policy.

Frequently asked questions

Do I need a reranker if I already have hybrid search?

Only if there is a gap between recall@50 and recall@5. Hybrid search improves what is in the candidate pool; a reranker improves what sits at the top of it. If your generator only ever sees five or six chunks, ordering is the entire game — measure the gap first, and skip the reranker if it is already small.

What value of k should I use in reciprocal rank fusion?

60 is the standard default and rarely worth tuning. Lower values make each retriever’s top result dominate the fusion; higher values flatten contributions across the whole ranked list. If you are going to tune anything, tune the per-retriever weights first — they encode a real assumption about whether your users type exact terms or describe intent.

Should I use HyDE for every query?

No. Hypothetical document embeddings help short, vague, conversational queries and hurt exact-lookup queries, where a fabricated answer moves the query vector away from the chunk that actually contains the identifier. Route it by query length and intent, and cache hypotheticals since traffic is heavily skewed toward a small set of repeated queries.

How large does my RAG eval set need to be?

150–300 labelled queries is enough to detect meaningful regressions, provided you label at chunk level and include roughly 10% unanswerable queries. Coverage of query types matters more than size: a 200-query set spanning lookups, comparisons, and multi-hop questions beats a 1000-query set of near-duplicates.

Conclusion

Production RAG is a retrieval engineering problem, not a prompting problem. Chunk on structure, keep tables atomic, and breadcrumb every chunk. Fuse BM25 with dense retrieval using reciprocal rank fusion, retrieving deep enough on both sides for the fusion to matter. Re-rank top-50 down to top-6 with a cross-encoder. Filter metadata before the search, not after it. Then prove all of it with a labelled eval set that includes unanswerable queries.

Do it in that order and each stage has a measurable effect you can defend. Skip to the end — rewriting queries over badly chunked documents — and you ship latency without accuracy. For the layer above this one, see evaluating AI models to pick the generator, and revisit embeddings and RAG for the embedding layer itself.

Related reading

• AI Embeddings Explained: Vectors, Similarity, and Building Your First RAG
• How to Choose a Vector Database for RAG
• Building a RAG Ingestion Pipeline: Crawling, Parsing, and Syncing
• Detecting and Reducing Hallucinations in Production LLM Apps
• Building a Streaming Chat UI in React: Patterns for SSE Responses
• Output Guardrails: Validating LLM Responses in Production
• Grounding LLM Answers with Web Search
• A/B Testing Prompts and Models in Production