{"id":114,"date":"2026-09-17T01:45:47","date_gmt":"2026-09-16T17:45:47","guid":{"rendered":"https:\/\/wp.qoraapi.com\/production-rag-architecture\/"},"modified":"2026-09-22T17:51:22","modified_gmt":"2026-09-22T09:51:22","slug":"production-rag-architecture","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/production-rag-architecture\/","title":{"rendered":"Building Production RAG: Chunking, Hybrid Search, and Re-Ranking"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Production RAG fails in three measurable places: chunks that split an answer across a boundary, pure vector search that misses exact identifiers, and no re-ranking, so a mediocre retriever feeds noise into a good model. Fix retrieval in that order \u2014 semantic chunking, BM25 plus vector fusion, cross-encoder re-ranking \u2014 before you touch the prompt.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is the follow-up to <a href=\"https:\/\/qoraapi.com\/blog\/ai-embeddings-rag\/\">embeddings and RAG<\/a>, which covers what embeddings are and how to call them. This article assumes you already have a working vector index and your answers are still wrong, and covers the four retrieval stages that close the gap.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why naive RAG fails in production<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A demo works because you wrote the five test questions yourself. Production breaks in four diagnosable ways.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Chunk boundaries cut answers in half.<\/strong> The retriever returns the chunk containing the question but not the chunk containing the answer. This is the most common cause of &#8220;the answer is in the docs but the bot says it isn&#8217;t&#8221; \u2014 and it stays invisible unless you inspect retrieved chunks, not just the final text.<\/li>\n<li><strong>Dense embeddings are lexically blind.<\/strong> A bi-encoder compresses rare tokens into a general region of the vector space, so <code>ERR_CONN_REFUSED_0x5<\/code>, SKU <code>A-4471-B<\/code>, and <code>invoice.total_cents<\/code> all land near semantically similar but wrong neighbours. Systems that &#8220;work for questions and fail for lookups&#8221; are almost always failing here.<\/li>\n<li><strong>Top-k dilutes signal.<\/strong> Bi-encoder similarity is not calibrated relevance: rank 1 means &#8220;least dissimilar&#8221;, not &#8220;correct&#8221;. Passing five chunks when one matters gives the model four opportunities to anchor on the wrong context, and content buried in the middle of a long context is used less reliably.<\/li>\n<li><strong>Nobody measured retrieval.<\/strong> Teams rewrite prompts for weeks while recall@10 sits at 0.5. The generator cannot fix a document that was never retrieved.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The operating rule: <strong>if recall@10 is below roughly 0.85, stop tuning the prompt.<\/strong> Prompt engineering cannot recover an answer that never entered the context window. Everything below is retrieval work.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Chunking strategies: set the size from your data, not from a blog post<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Two measurements decide chunk size for you. First, the <em>answer span<\/em>: the median character length of the passage a correct answer actually needs. FAQ corpora need 200\u2013400 characters; API references with multi-step procedures need 800\u20131500. Second, <em>query specificity<\/em>: exact-lookup queries want smaller, more precise chunks, while &#8220;explain the architecture&#8221; queries want larger ones. Sample 50 real queries, label the minimum passage that answers each, and take the 75th percentile of that length. That number is your target size.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Fixed-size splitting with overlap is predictable and cheap, but it cuts mid-thought. Recursive or structural splitting \u2014 headings, then paragraphs, then sentences \u2014 is the correct default. Semantic chunking, where you embed sentences and cut where consecutive-sentence similarity drops below a threshold, sounds better than it usually is: it produces wildly variable chunk sizes, which breaks your embedding cost model and can emit 40-token fragments that carry no retrievable signal. If you use it, clamp it to a band (say 200\u20131200 characters) and fall back to recursive splitting inside those bounds.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Overlap deserves more scepticism than it gets. Overlap does not add information; it duplicates it, inflating storage and embedding cost and producing near-duplicate hits that crowd out diversity in your top-k. Apply overlap only at paragraph seams, never inside tables or code, and deduplicate by content hash at retrieval time.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Content type<\/th><th>Splitter<\/th><th>Size \/ overlap<\/th><th>Why<\/th><\/tr><\/thead><tbody><tr><td>Prose docs, articles<\/td><td>Recursive on headings then paragraphs<\/td><td>600\u20131000 chars, 10\u201315% overlap<\/td><td>Preserves argument flow; overlap only at paragraph seams<\/td><\/tr><tr><td>API reference, config<\/td><td>Structural + heading breadcrumb<\/td><td>300\u2013700 chars, no overlap<\/td><td>Each endpoint is self-contained; the breadcrumb restores the context the size removes<\/td><\/tr><tr><td>Tables, spec matrices<\/td><td>Atomic, rows flattened to key:value<\/td><td>One table per chunk set, no overlap<\/td><td>A data row without its header row is unrecoverable<\/td><\/tr><tr><td>Source code<\/td><td>AST: function and class boundaries<\/td><td>One symbol per chunk, signature prepended<\/td><td>The retrieval target is a symbol, not a byte range<\/td><\/tr><tr><td>FAQ, support tickets<\/td><td>One Q&amp;A pair per chunk<\/td><td>150\u2013400 chars, no overlap<\/td><td>Matches the shape of the incoming query distribution<\/td><\/tr><tr><td>Contracts, policies<\/td><td>Clause-level, by numbering<\/td><td>Clause boundaries, no overlap<\/td><td>Citations must map back to a clause number a human can verify<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Two structural moves pay for themselves immediately. <strong>Never split a table<\/strong> \u2014 and flatten each row into <code>column: value<\/code> lines so column names become lexically searchable, which is exactly what the hybrid stage below needs. <strong>Prepend the heading path<\/strong> to every chunk: a chunk reading &#8220;Set the timeout to 30&#8221; is nearly useless, while &#8220;Payments API &gt; Retries &gt; Configuration \u2014 Set the timeout to 30&#8221; is retrievable by three different query phrasings. Both are cheap to implement and both are pure retrieval-quality gains.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import re\n\ndef split_sections(md: str):\n    \"\"\"Split markdown on headings, keeping code fences and tables atomic.\"\"\"\n    parts, cur, path, in_fence = [], [], [], False\n    for line in md.splitlines():\n        if line.startswith(\"```\"):\n            in_fence = not in_fence\n        if not in_fence and re.match(r\"^#{1,6}\\s\", line):\n            if cur:\n                parts.append((\"\\n\".join(path), \"\\n\".join(cur).strip()))\n            level = len(line) - len(line.lstrip(\"#\"))\n            path = path[: level - 1] + [line.lstrip(\"# \").strip()]\n            cur = [line]\n        else:\n            cur.append(line)\n    if cur:\n        parts.append((\"\\n\".join(path), \"\\n\".join(cur).strip()))\n\n    # Breadcrumb every chunk so short splits stay retrievable.\n    return [{\"id\": str(i), \"breadcrumb\": head, \"text\": f\"{head}\\n{body}\"}\n            for i, (head, body) in enumerate(parts) if body]\n\n\ndef flatten_table(header: str, rows: list) -> str:\n    \"\"\"Turn a markdown table into key:value lines so BM25 can hit column names.\"\"\"\n    cols = [c.strip() for c in header.strip(\"|\").split(\"|\")]\n    out = []\n    for row in rows:\n        cells = [c.strip() for c in row.strip(\"|\").split(\"|\")]\n        out.append(\"; \".join(f\"{c}: {v}\" for c, v in zip(cols, cells) if v))\n    return \"\\n\".join(out)\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Hybrid search: BM25 plus vectors, fused with reciprocal rank fusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">BM25 scores exact term overlap with inverse document frequency weighting, which makes it unbeatable on rare tokens: error codes, function names, part numbers, version strings. Dense retrieval handles paraphrase and synonymy, where BM25 scores zero because the words differ. They fail in opposite directions, so the union is strictly better than either \u2014 and hybrid retrieval is the highest-value single change most RAG systems can make.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The fusion problem is that the two score scales are incomparable. BM25 scores are unbounded and depend on corpus statistics; cosine similarities are bounded and depend on the embedding model. Min-max normalising them per query looks reasonable and is unstable in practice, because the normalisation is driven by whatever happened to be in that query&#8217;s result set.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Reciprocal Rank Fusion (RRF)<\/strong> avoids the problem entirely by discarding the scores and fusing the ranks: <code>score(d) = \u03a3 w<sub>r<\/sub> \/ (k + rank<sub>r<\/sub>(d))<\/code>. The constant <code>k<\/code>, usually 60, damps the influence of the very top ranks; smaller values make each retriever&#8217;s top-1 dominate, larger values flatten contributions across the list. Sixty is a robust default and rarely worth tuning before you have an eval set \u2014 tune the per-retriever weights <code>w<sub>r<\/sub><\/code> first, since that is where real bias lives.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One detail decides whether RRF works at all: <strong>how deep you retrieve from each side<\/strong>. Fuse the top 50 from each retriever, not the top 10. A document ranked 30th by both retrievers is a strong relevance signal that never enters a shallow pool.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import numpy as np\nfrom collections import defaultdict\n\n\ndef rrf_fuse(rankings, k=60, weights=None):\n    \"\"\"rankings: list of ranked id lists (best first). Returns id -> fused score.\"\"\"\n    weights = weights or [1.0] * len(rankings)\n    fused = defaultdict(float)\n    for ranking, w in zip(rankings, weights):\n        for rank, doc_id in enumerate(ranking, start=1):\n            fused[doc_id] += w \/ (k + rank)\n    return dict(sorted(fused.items(), key=lambda kv: -kv[1]))\n\n\ndef hybrid_search(query, chunks, bm25, faiss_index, embed,\n                  depth=50, w_lex=1.0, w_dense=1.0):\n    \"\"\"BM25 + dense retrieval, fused by reciprocal rank fusion.\"\"\"\n    ids = [c[\"id\"] for c in chunks]\n\n    # 1. lexical side - exact terms, rare identifiers, column names\n    lex_scores = bm25.get_scores(query.lower().split())\n    lexical = [ids[i] for i in np.argsort(-lex_scores)[:depth]]\n\n    # 2. dense side - paraphrase and synonymy\n    qv = np.asarray([embed(query)], dtype=\"float32\")\n    _, idx = faiss_index.search(qv, depth)\n    dense = [ids[i] for i in idx[0] if i != -1]\n\n    # 3. rank-based fusion - no score normalisation needed\n    fused = rrf_fuse([lexical, dense], k=60, weights=[w_lex, w_dense])\n    return list(fused)[:depth]\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Use <code>w_lex=2.0<\/code> when your corpus is identifier-heavy (logs, code, SKUs, legal citations) and <code>w_dense=2.0<\/code> when queries are conversational and users rarely type exact terms.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Re-ranking: take top-K from hybrid, return top-N to the model<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A bi-encoder encodes the query and the document independently, so it can never model the interaction between their terms. A <strong>cross-encoder<\/strong> concatenates query and document and scores them jointly, which is substantially more accurate \u2014 and costs one forward pass per candidate. That cost structure is exactly why it belongs in a second stage: you cannot run it over a corpus, but you can run it over 50 candidates.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def retrieve_and_rerank(query, chunks, bm25, faiss_index, embed, rerank,\n                        k_retrieve=50, k_final=6):\n    \"\"\"Stage 1: hybrid recall@50. Stage 2: cross-encoder precision@6.\"\"\"\n    candidates = hybrid_search(query, chunks, bm25, faiss_index, embed,\n                               depth=k_retrieve)\n    by_id = {c[\"id\"]: c for c in chunks}\n    docs = [by_id[cid][\"text\"] for cid in candidates]\n\n    # One batched call - per-document HTTP overhead dominates at K=50.\n    result = rerank(query=query, documents=docs, top_n=k_final)\n\n    # Map reranker positions back to the original chunk objects.\n    return [by_id[candidates[r[\"index\"]]] for r in result[\"results\"]]\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Three things matter more than the choice of reranker:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Re-ranking changes your optimal chunk size.<\/strong> With a reranker in the pipeline you can retrieve small, precise chunks \u2014 better lexical match, less noise in the vector \u2014 and then expand to the parent chunk before generation. This &#8220;small-to-big&#8221; pattern is often a larger quality win than the reranker itself, and it only becomes safe once a cross-encoder is filtering the pool.<\/li>\n<li><strong>Cap K at the knee.<\/strong> Cross-encoder latency grows roughly linearly in the number of candidates. K=50 is usually the knee; pushing to 200 buys a point or two of recall for several times the rerank latency.<\/li>\n<li><strong>Watch for the flip.<\/strong> If reranking consistently demotes your top BM25 hit, your lexical weight is too high \u2014 you are promoting chunks that match surface terms without answering the question.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Query rewriting and metadata filtering<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Hypothetical document embeddings (HyDE).<\/strong> Instead of embedding the user&#8217;s query, ask a small model to write a short hypothetical answer and embed <em>that<\/em>. The intuition is sound: an answer-shaped passage sits closer in embedding space to real answer chunks than a six-word question does. The routing rule matters more than the technique \u2014 enable HyDE when the query is under about five tokens or is an open &#8220;how\/why&#8221; question, and disable it for exact-lookup queries, where a fabricated hypothetical answer actively pulls the query vector away from the correct chunk. Because it adds a full generation call, cache the hypotheticals: query traffic is heavily skewed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Multi-query expansion<\/strong> \u2014 three paraphrases, retrieve for each, fuse with RRF \u2014 improves recall and triples retrieval cost. Use it on the recall-critical path only.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Metadata filtering<\/strong> is where production systems break quietly. Four rules:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Pre-filter, do not post-filter.<\/strong> Dropping chunks after ranking fails badly on selective filters: if a tenant is 1% of the corpus, a 50-document candidate pool contains roughly half of one of their documents. Apply filters inside the ANN search, or over-fetch by at least 1\/selectivity.<\/li>\n<li><strong>Design the facets you will actually filter on:<\/strong> tenant or workspace id, document type, source system, effective date or version, and access level. Enforce them at the index layer, never by asking the model to ignore content.<\/li>\n<li><strong>Version and date filters are the cheapest fix for stale answers.<\/strong> &#8220;Latest policy&#8221; without a date filter will surface a superseded document that is semantically identical to the current one \u2014 the model has no way to prefer the newer text.<\/li>\n<li><strong>Access control is a hard filter, not a ranking signal.<\/strong> A post-filter that removes unauthorised chunks after ranking still leaks their existence through scores and ordering, and can be defeated by increasing k. Retrieval is a security boundary.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Evaluation: does retrieval actually help?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Evaluate the two layers separately, or you will never know which one to fix. <strong>Retrieval quality<\/strong> is measured against a labelled set of query-to-relevant-chunk mappings: recall@k tells you whether the answer is even in the pool, which is the ceiling on end-to-end accuracy, while MRR and nDCG@10 tell you whether it is near the top, which is what re-ranking and context ordering control. <strong>Answer quality<\/strong> is measured on the generated text: groundedness (does every claim map to a cited chunk?), citation precision, and abstention correctness.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The non-obvious requirement: <strong>include unanswerable queries.<\/strong> Most RAG eval sets contain only questions the corpus can answer, so they cannot detect the failure mode that destroys user trust fastest \u2014 a confident answer assembled from irrelevant context. Aim for roughly 10% unanswerable queries and track the leak rate.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Metric<\/th><th>Stage<\/th><th>What it catches<\/th><th>Practical target<\/th><\/tr><\/thead><tbody><tr><td>recall@50<\/td><td>Retrieval (hybrid)<\/td><td>Answer absent from the candidate pool \u2014 the accuracy ceiling<\/td><td>&gt; 0.90<\/td><\/tr><tr><td>recall@5<\/td><td>Post-rerank<\/td><td>Whether re-ranking actually improved ordering<\/td><td>&gt; 0.75<\/td><\/tr><tr><td>MRR \/ nDCG@10<\/td><td>Ranking<\/td><td>Relevant chunk buried under near-duplicate noise<\/td><td>MRR &gt; 0.70<\/td><\/tr><tr><td>Citation precision<\/td><td>Generation<\/td><td>Model citing chunks that do not support the claim<\/td><td>&gt; 0.90<\/td><\/tr><tr><td>Abstention leak rate<\/td><td>End to end<\/td><td>Confident answers built from irrelevant context<\/td><td>&lt; 5%<\/td><\/tr><tr><td>p95 latency<\/td><td>System<\/td><td>Whether the extra stages are actually shippable<\/td><td>Defined by your budget<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">An offline harness is a hundred lines. Label at <em>chunk<\/em> level, not document level \u2014 document-level labels hide chunking failures, which is precisely what you are trying to detect.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def evaluate(retrieve, dataset, ks=(1, 5, 10, 50)):\n    \"\"\"dataset: [{\"query\": str, \"relevant\": set[chunk_id], \"answerable\": bool}]\"\"\"\n    hits = {k: 0 for k in ks}\n    rr_sum, leaks = 0.0, 0\n    answerable = [row for row in dataset if row[\"answerable\"]]\n    unanswerable = [row for row in dataset if not row[\"answerable\"]]\n\n    for row in answerable:\n        ranked = [c[\"id\"] for c in retrieve(row[\"query\"])]\n        first = next((i for i, cid in enumerate(ranked, 1)\n                      if cid in row[\"relevant\"]), None)\n        rr_sum += 1.0 \/ first if first else 0.0\n        for k in ks:\n            hits[k] += bool(set(ranked[:k]) &amp; row[\"relevant\"])\n\n    for row in unanswerable:\n        ranked = [c[\"id\"] for c in retrieve(row[\"query\"])]\n        leaks += bool(ranked and ranked[0] not in row[\"relevant\"])\n\n    n = len(answerable)\n    report = {f\"recall@{k}\": round(hits[k] \/ n, 3) for k in ks}\n    report[\"mrr\"] = round(rr_sum \/ n, 3)\n    report[\"leak_rate\"] = round(leaks \/ max(len(unanswerable), 1), 3)\n    return report\n\n\n# Ablation gate: run before and after every index or prompt change.\n# baseline             recall@5 0.61  mrr 0.58\n# + semantic chunking  recall@5 0.72  mrr 0.66\n# + hybrid + rrf       recall@5 0.79  mrr 0.71\n# + cross-encoder      recall@5 0.86  mrr 0.83\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Build the labelled set from real query logs plus every production failure, verbatim. 150\u2013300 queries is enough to detect meaningful regressions if the set spans query <em>types<\/em> \u2014 lookups, comparisons, multi-hop questions \u2014 because coverage beats raw size. Treat recall@10 as a regression gate in CI. For the answer-quality layer, the same discipline that makes model outputs gradeable applies: fixed schemas and deterministic scoring, as covered in our guide to <a href=\"https:\/\/qoraapi.com\/blog\/ai-structured-outputs-json-mode\/\">structured outputs<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Cost and latency of the extra stages<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Latency in a RAG pipeline is additive. For an interactive Q&amp;A path the shape is stable: query embedding is one API round trip; BM25 runs in-process in single-digit milliseconds; ANN search over 10<sup>5<\/sup>\u201310<sup>6<\/sup> vectors is a few tens of milliseconds; RRF fusion is arithmetic; cross-encoder re-ranking over K=50 is the dominant added stage; generation is usually the largest component of all.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Costs follow the same asymmetry. Re-ranking is priced per document, so it scales with K times query volume \u2014 but you are scoring a few dozen short passages, which typically makes it a small fraction of the token cost of generating the answer. HyDE, by contrast, adds an entire generation call, the same order of magnitude as the answer itself. <strong>Re-ranking is cheap precision; rewriting is expensive recall.<\/strong> That one distinction explains most of the design decisions below.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Add hybrid search unconditionally.<\/strong> BM25 is in-process, RRF is arithmetic, and the only added cost is a slightly larger candidate pool.<\/li>\n<li><strong>Add a reranker when recall@50 is much higher than recall@5.<\/strong> That gap is precisely the precision the cross-encoder recovers. If recall@5 already equals recall@50, your retriever is already precise and re-ranking buys latency for nothing.<\/li>\n<li><strong>Add HyDE only if a meaningful share of traffic is short and vague.<\/strong> Route it by query length and intent instead of applying it globally, and cache the generated hypotheticals.<\/li>\n<li><strong>Add multi-query expansion last, and only where recall is critical.<\/strong> It is the only stage that multiplies retrieval cost by design.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The sequencing matters as much as the stages. Fix chunking first \u2014 it is free and delivers the largest single jump. Then add BM25 and RRF, which are nearly free. Then re-ranking, which costs a little. Rewriting comes last. Teams that start with query rewriting pay the most and improve the least, because they are rewriting queries against a corpus that was chunked badly. Serving embeddings and reranking through one OpenAI-compatible endpoint keeps this from turning into a vendor-management problem; <a href=\"https:\/\/qoraapi.com\/\" target=\"_blank\" rel=\"noopener\">qoraapi.com<\/a> exposes embedding and rerank models behind a single API key, so the pipeline above stays one credential and one retry policy.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Do I need a reranker if I already have hybrid search?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Only if there is a gap between recall@50 and recall@5. Hybrid search improves what is in the candidate pool; a reranker improves what sits at the top of it. If your generator only ever sees five or six chunks, ordering is the entire game \u2014 measure the gap first, and skip the reranker if it is already small.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What value of k should I use in reciprocal rank fusion?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">60 is the standard default and rarely worth tuning. Lower values make each retriever&#8217;s top result dominate the fusion; higher values flatten contributions across the whole ranked list. If you are going to tune anything, tune the per-retriever weights first \u2014 they encode a real assumption about whether your users type exact terms or describe intent.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should I use HyDE for every query?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. Hypothetical document embeddings help short, vague, conversational queries and hurt exact-lookup queries, where a fabricated answer moves the query vector away from the chunk that actually contains the identifier. Route it by query length and intent, and cache hypotheticals since traffic is heavily skewed toward a small set of repeated queries.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How large does my RAG eval set need to be?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">150\u2013300 labelled queries is enough to detect meaningful regressions, provided you label at chunk level and include roughly 10% unanswerable queries. Coverage of query types matters more than size: a 200-query set spanning lookups, comparisons, and multi-hop questions beats a 1000-query set of near-duplicates.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Production RAG is a retrieval engineering problem, not a prompting problem. Chunk on structure, keep tables atomic, and breadcrumb every chunk. Fuse BM25 with dense retrieval using reciprocal rank fusion, retrieving deep enough on both sides for the fusion to matter. Re-rank top-50 down to top-6 with a cross-encoder. Filter metadata before the search, not after it. Then prove all of it with a labelled eval set that includes unanswerable queries.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Do it in that order and each stage has a measurable effect you can defend. Skip to the end \u2014 rewriting queries over badly chunked documents \u2014 and you ship latency without accuracy. For the layer above this one, see <a href=\"https:\/\/qoraapi.com\/blog\/evaluate-benchmark-ai-models\/\">evaluating AI models<\/a> to pick the generator, and revisit <a href=\"https:\/\/qoraapi.com\/blog\/ai-embeddings-rag\/\">embeddings and RAG<\/a> for the embedding layer itself.<\/p>\n\n\n\n\n<h3 class=\"wp-block-heading\">Related reading<\/h3>\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-embeddings-rag\/\">AI Embeddings Explained: Vectors, Similarity, and Building Your First RAG<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/vector-database-selection\/\">How to Choose a Vector Database for RAG<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/rag-ingestion-pipeline\/\">Building a RAG Ingestion Pipeline: Crawling, Parsing, and Syncing<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/detect-reduce-hallucinations\/\">Detecting and Reducing Hallucinations in Production LLM Apps<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/streaming-chat-ui-react\/\">Building a Streaming Chat UI in React: Patterns for SSE Responses<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/llm-output-guardrails\/\">Output Guardrails: Validating LLM Responses in Production<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/grounding-llm-web-search\/\">Grounding LLM Answers with Web Search<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ab-testing-prompts-models\/\">A\/B Testing Prompts and Models in Production<\/a><\/li><\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>Production RAG needs more than vector search. Learn semantic chunking, BM25+vector hybrid retrieval with reciprocal rank fusion, and cross-encoder re-ranking.<\/p>\n","protected":false},"author":1,"featured_media":113,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,6,9,7],"class_list":["post-114","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-api","tag-ai-api","tag-api-gateway","tag-developer-tools","tag-developers"],"_links":{"self":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/114","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/comments?post=114"}],"version-history":[{"count":3,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/114\/revisions"}],"predecessor-version":[{"id":317,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/114\/revisions\/317"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media\/113"}],"wp:attachment":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media?parent=114"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/categories?post=114"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/tags?post=114"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}