{"id":162,"date":"2026-09-20T02:27:36","date_gmt":"2026-09-19T18:27:36","guid":{"rendered":"https:\/\/wp.qoraapi.com\/rag-ingestion-pipeline\/"},"modified":"2026-09-20T03:53:57","modified_gmt":"2026-09-19T19:53:57","slug":"rag-ingestion-pipeline","status":"publish","type":"post","link":"https:\/\/qoraapi.com\/blog\/rag-ingestion-pipeline\/","title":{"rendered":"Building a RAG Ingestion Pipeline: Crawling, Parsing, and Syncing"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A RAG ingestion pipeline is an ETL job in three stages: crawl sources into raw documents, parse and normalize them into clean text plus metadata, then chunk, embed, and sync into a vector index \u2014 incrementally, using content hashes for updates and tombstones for deletes. Retrieval quality is permanently capped by what this pipeline emits.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Everything downstream operates on the text your parser produced. This guide covers the ingest side: connectors, parsing, chunk storage, incremental sync, ACL propagation, and embedding economics.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why ingestion is where RAG projects quietly fail<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The symptom is always the same. A demo works on five hand-picked PDFs; three weeks later users say the assistant is confidently wrong, and the team reranks, swaps embedding models, and tunes <code>top_k<\/code>. Nothing fixes it, because the defect is upstream: a scanned contract parsed to an empty string, a wiki sidebar repeated in every chunk, a deleted policy still being cited.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The economics are asymmetric. A retrieval bug affects one query. An ingestion bug affects <em>every query that touches that document, forever<\/em> \u2014 and it stays invisible in your metrics, because retrieval is working correctly: it faithfully returns the garbage you stored.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So treat parsing as a validated step with an explicit contract. Two rules carry most of the weight:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Measure parse yield.<\/strong> Compute tokens-per-page for every document and set a floor \u2014 say 200 tokens per page for prose PDFs. Anything below it goes to quarantine for human review, never into the index.<\/li>\n<li><strong>Fail loudly, never silently.<\/strong> A parser returning 300 tokens from a 40-page report has &#8220;succeeded&#8221; and poisoned your index. Assert on page count, heading count, and table count; fail the document when the ratio collapses.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Keep the raw artifacts. An index is a build artifact \u2014 you should be able to rebuild it from raw documents plus a parse-config version. Teams that discard originals can never fix a parser bug retroactively or change embedding models cheaply.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Sources and connectors: what each one actually needs<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A connector is not &#8220;download the file.&#8221; Each source class carries different metadata, deletion semantics, and failure modes:<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Source<\/th><th>What you get<\/th><th>What the connector must handle<\/th><\/tr><\/thead><tbody><tr><td>Docs in Git (Markdown \/ MDX)<\/td><td>Text + frontmatter<\/td><td>Read from the repo, not the rendered site \u2014 you keep history, frontmatter, and a commit SHA to version chunks with. Deletions arrive as a git diff.<\/td><\/tr><tr><td>Wikis (Confluence, Notion)<\/td><td>Block JSON or HTML<\/td><td>Pagination, nested child pages, per-page permission lists, and an <code>updated_at<\/code> that actually changes. Strip editor chrome.<\/td><\/tr><tr><td>PDFs (contracts, specs, scans)<\/td><td>Binary; may lack a text layer<\/td><td>Layout-aware parse with reading-order reconstruction, OCR fallback for image-only pages, page numbers preserved for citations.<\/td><\/tr><tr><td>Databases<\/td><td>Rows<\/td><td>Incremental by an <code>updated_at<\/code>\/id watermark. One text projection per row \u2014 never dump whole tables. Mirror row permissions into a groups column at ingest.<\/td><\/tr><tr><td>SaaS APIs (tickets, issues)<\/td><td>Nested JSON<\/td><td>Separate description from comment thread, filter closed\/resolved noise if users only search live work, redact PII before embedding.<\/td><\/tr><tr><td>Public web pages<\/td><td>HTML with chrome<\/td><td>Respect robots.txt and crawl rate, extract the main content region only, record fetch time \u2014 web pages have no <code>updated_at<\/code>.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The governing rule: <strong>prefer the source of truth that carries the metadata you need.<\/strong> Rendering a docs site to HTML throws away git history and ACLs; pulling the same content through the repository API keeps both.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Parsing: layout, structure, and tables<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Text extraction is solved only for plain text. Real corpora pose three problems at once.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Reading order<\/strong> is where naive PDF extraction dies. Multi-column layouts, sidebars, and footers get interleaved into nonsense. A layout-aware extractor reconstructs blocks by position and drops repeated header\/footer bands. Check for a text layer first: a page yielding fewer than ~50 characters is almost certainly an image needing OCR \u2014 for that page only.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Structure preservation<\/strong> means keeping the heading hierarchy in the extracted text, because you need it for chunking and cannot rebuild it later. Convert headings to a marked form (<code>#<\/code>, <code>##<\/code>) at parse time so the chunker splits on document structure, not a fixed token count.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Tables<\/strong> must be emitted as tables \u2014 Markdown or HTML \u2014 never flattened prose. Flattening destroys the row-column binding, so <code>Q1 | 12% | 8%<\/code> becomes an unattributable string of numbers. When a table spans chunks, prepend the header row to every slice.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here is a working parse-and-normalize step with validation built in:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import hashlib, re\nfrom dataclasses import dataclass, field\nfrom bs4 import BeautifulSoup\nimport fitz  # PyMuPDF\n\nMIN_CHARS_PER_PAGE = 50\nDROP = {\"script\", \"style\", \"nav\", \"footer\", \"aside\", \"form\", \"noscript\"}\nHEADING = re.compile(r\"^h[1-6]$\")\n\n@dataclass\nclass Doc:\n    doc_id: str\n    source: str\n    uri: str\n    text: str\n    metadata: dict = field(default_factory=dict)\n    content_hash: str = \"\"\n\ndef normalize(text: str) -&gt; str:\n    text = text.replace(\"\\u00ad\", \"\")             # soft hyphens\n    text = re.sub(r\"[ \\t]+\", \" \", text)\n    text = re.sub(r\"\\n{3,}\", \"\\n\\n\", text)\n    text = re.sub(r\"(?m)^\\s*\\d+\\s*$\", \"\", text)   # bare page numbers\n    return text.strip()\n\ndef parse_pdf(path: str) -&gt; Doc:\n    pdf, pages, ocr_pages = fitz.open(path), [], 0\n    for i, page in enumerate(pdf):\n        raw = page.get_text(\"text\")\n        if len(raw.strip()) &lt; MIN_CHARS_PER_PAGE:\n            ocr_pages += 1\n            raw = ocr_page(page)                  # your OCR adapter\n        pages.append(f\"[page {i+1}]\\n{normalize(raw)}\")\n    # fail loudly: an image-only PDF whose OCR we cannot vouch for\n    if ocr_pages &gt; len(pages) * 0.8:\n        raise ValueError(f\"{path}: image-only, OCR quality unverified\")\n    return Doc(doc_id=path, source=\"pdf\", uri=path,\n               text=\"\\n\\n\".join(pages),\n               metadata={\"pages\": len(pages), \"ocr_pages\": ocr_pages})\n\ndef parse_html(html: str, uri: str) -&gt; Doc:\n    soup = BeautifulSoup(html, \"lxml\")\n    for tag in soup.find_all(DROP):\n        tag.decompose()\n    root = soup.find(\"main\") or soup.find(\"article\") or soup.body or soup\n    # keep structure: mark headings so the chunker can split on them\n    for h in root.find_all(HEADING):\n        h.insert_before(f\"\\n\\n{'#' * int(h.name[1])} \")\n    return Doc(doc_id=uri, source=\"html\", uri=uri,\n               text=normalize(root.get_text(\"\\n\")),\n               metadata={\"title\": (soup.title.string or \"\").strip()})\n\ndef with_hash(doc: Doc, parse_cfg: str = \"v3\") -&gt; Doc:\n    # hash NORMALIZED text + parse config, not raw bytes: a re-exported\n    # PDF that only changes timestamps must not trigger re-embedding.\n    payload = f\"{parse_cfg}\\n{doc.text}\"\n    doc.content_hash = hashlib.sha256(payload.encode()).hexdigest()\n    return doc<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two details matter most. The hash covers normalized text plus a <code>parse_cfg<\/code> version, so a parser upgrade deliberately invalidates everything while a no-op file change does not. And the OCR guard raises instead of returning a plausible stub \u2014 preventing the most common silent failure in document RAG.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Chunking at ingest vs at query time<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Most chunking advice conflates two decisions. Separate them and the design becomes obvious.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ingest-time chunking<\/strong> decides what you <em>embed<\/em> and what you <em>store<\/em> \u2014 not the same unit. Embed small children (200\u2013500 tokens) so the vector matches a short query precisely; store the larger parent section (1,000\u20132,000 tokens) keyed by <code>parent_id<\/code>.<\/li>\n<li><strong>Query-time assembly<\/strong> decides what the model <em>sees<\/em>: retrieve the children, then expand each hit to its parent \u2014 &#8220;small-to-big&#8221;. Precise matching and full context in one request, without guessing a single chunk size that satisfies both.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Two techniques belong to the ingest side and cost nothing at query time:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Contextual prefixes.<\/strong> Prepend the document title and heading path to each child before embedding: <code>Billing API &gt; Rate limits &gt; Burst allowance<\/code>. A 200-token chunk is often ambiguous alone; the breadcrumb stops it colliding with every other &#8220;\u2026allowance&#8221; paragraph in the corpus.<\/li>\n<li><strong>Structural boundaries over fixed windows.<\/strong> Split on headings, list items, and table rows first; fall back to a token window only when one section exceeds the limit. A fixed 512-token window slices tables in half and splits procedures between steps 4 and 5. Overlap of 10\u201315% at that boundary is a sane default.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Everything on the retrieval side \u2014 hybrid search, reranking, fusion, context budgeting \u2014 belongs to our guide on <a href=\"https:\/\/qoraapi.com\/blog\/production-rag-architecture\/\">production RAG<\/a>. Ingestion owns text quality, chunk identity, and metadata; retrieval owns ranking and assembly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Incremental sync and change detection<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Full re-ingestion is fine at 5,000 chunks and ruinous at 5 million. Incremental sync classifies each document into a change type and takes the cheapest correct action:<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table class=\"has-fixed-layout\"><thead><tr><th>Change type<\/th><th>Signal<\/th><th>Action<\/th><\/tr><\/thead><tbody><tr><td>New document<\/td><td><code>doc_id<\/code> absent from the index<\/td><td>Parse \u2192 chunk \u2192 embed \u2192 upsert<\/td><\/tr><tr><td>Content updated<\/td><td><code>content_hash<\/code> differs<\/td><td>Re-parse; re-embed only chunks whose own hash changed; delete orphaned chunk ids<\/td><\/tr><tr><td>Metadata-only change (title, ACL)<\/td><td><code>metadata_hash<\/code> differs, <code>content_hash<\/code> unchanged<\/td><td>Update metadata in place \u2014 <strong>no re-embedding<\/strong><\/td><\/tr><tr><td>Deleted at source<\/td><td>Absent from a full source listing, or <code>deleted_at<\/code> set<\/td><td><strong>Tombstone<\/strong>: mark deleted, remove vectors, exclude from search<\/td><\/tr><tr><td>Moved or renamed<\/td><td>Same <code>content_hash<\/code>, new URI<\/td><td>Update <code>uri<\/code> and metadata only<\/td><\/tr><tr><td>Source unreachable<\/td><td>Connector error or timeout<\/td><td>Do nothing \u2014 never tombstone on a fetch failure<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The row most pipelines get wrong is deletion. Upsert-only ingestion means deletes never propagate, so deprecated policies and removed customer data stay retrievable \u2014 and get cited with full confidence. Two detection strategies trade off differently:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>CDC \/ <code>deleted_at<\/code>:<\/strong> cheap and near-real-time, but only if the source exposes deletions. Many do not.<\/li>\n<li><strong>Full-ID reconciliation:<\/strong> list every source id, diff against the index, tombstone the difference. Expensive but authoritative \u2014 the only approach that catches documents deleted while your connector was down.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Run reconciliation on a cadence \u2014 daily for high-churn sources, weekly for stable ones. Implement tombstones as <em>soft<\/em> deletes: set <code>deleted: true<\/code> plus <code>deleted_at<\/code>, filter them at query time, purge after a retention window. That makes a bad connector run reversible instead of catastrophic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Watermark sync has two traps. Subtract a safety lag (5\u201315 minutes) from the watermark, because transactions commit out of order and clocks skew; without it you silently skip rows. And use a composite <code>(updated_at, id)<\/code> cursor rather than a timestamp alone, or rows sharing a timestamp get skipped on ties. Commit the watermark only after the write succeeds:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def sync_table(conn, index, cursor):\n    rows = conn.execute(\n        \"\"\"select id, body, updated_at from docs\n           where (updated_at, id) &gt; (%s, %s)\n             and updated_at &lt; now() - interval '10 minutes'\n           order by updated_at, id limit 500\"\"\",\n        (cursor[\"updated_at\"], cursor[\"id\"])).fetchall()\n\n    for row in rows:\n        doc = with_hash(parse_row(row))          # content_hash + metadata_hash\n        if index.get_hash(doc.doc_id) == doc.content_hash:\n            continue                             # no-op: costs zero embeddings\n        index.upsert(embed_chunks(doc))          # only changed chunks embed\n\n    if rows:\n        index.commit()\n        cursor.update(updated_at=rows[-1].updated_at, id=rows[-1].id)\n    return len(rows)\n\ndef reconcile_deletes(conn, index):\n    live = {r.id for r in conn.execute(\"select id from docs\")}\n    stale = index.list_doc_ids() - live\n    index.tombstone(stale)                       # soft delete, purge later\n    return len(stale)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Note what the watermark cannot do: it cannot see deletions. Watermarks handle updates; reconciliation handles deletes. You need both.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Metadata and permissions: an index without ACLs leaks data<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A vector store has no row-level security by default. The moment you ingest an HR policy, a private ticket, or a restricted repository, you have created a shadow copy of your most sensitive content behind a single API key. This is the highest-severity failure mode in the pipeline, and it is entirely an ingestion problem.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The pattern that works: <strong>denormalize ACLs at ingest, filter at query time, and let the vector store enforce it<\/strong>.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Store permission fields on every chunk \u2014 <code>acl_groups: [\"eng\", \"sre\"]<\/code> \u2014 copied from the parent at ingest. Denormalized, because filtering happens per chunk.<\/li>\n<li>Pass the filter into the ANN search itself, built from the caller&#8217;s verified identity: <code>filter={\"acl_groups\": {\"$in\": user_groups}}<\/code>. The store never considers vectors the caller cannot see.<\/li>\n<li><strong>Never retrieve-then-filter.<\/strong> Fetching <code>top_k=10<\/code> and dropping 8 unauthorized hits gives a worse answer, wastes tokens, and turns one missing filter into a data breach.<\/li>\n<li>Treat ACL changes as content changes \u2014 the metadata-only row in the sync table. Update in place, skip the embedding call.<\/li>\n<li>Multi-tenant deployments need a per-tenant namespace <em>and<\/em> a per-tenant filter. The namespace bounds blast radius and query cost; the filter is the security control.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Where a database&#8217;s permission model cannot be expressed as groups, resolve it at ingest with a join that materializes a groups column per row. Evaluating a live model at query time puts a database round-trip inside your search path and asks the vector store to enforce authorization it cannot see. Record the ACL snapshot version on every chunk so you can answer &#8220;why did this user see that document in March.&#8221;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Cost and rate limits at scale<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Backfilling 500,000 chunks at roughly 400 tokens each is about 200 million tokens of embedding work. Sent synchronously at a few dozen requests per second, that is days of wall clock and a permanent stream of 429s. The methodology that keeps it manageable:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Batch the payload.<\/strong> Embedding endpoints accept arrays, so send many chunks per request. For an initial backfill or re-embed, route it through <a href=\"https:\/\/qoraapi.com\/blog\/batch-ai-api-processing\/\">batch AI APIs<\/a> where latency does not matter \u2014 the discount is substantial and throughput far higher.<\/li>\n<li><strong>Split hot and cold paths.<\/strong> Newly changed documents must be searchable in minutes and go through a small synchronous pool; backfills and model migrations go through the batch path. One queue for both means a backfill starves your freshness SLA.<\/li>\n<li><strong>Cap concurrency, then back off.<\/strong> Start at 4\u20138 in-flight requests, add exponential backoff with jitter on 429, and honor <code>Retry-After<\/code> \u2014 see our <a href=\"https:\/\/qoraapi.com\/blog\/ai-api-rate-limits-429-errors\/\">rate-limit handling guide<\/a>. A retry must never re-embed chunks that already committed.<\/li>\n<li><strong>Make jobs idempotent.<\/strong> Key every job by <code>(chunk_id, embed_model, parse_cfg)<\/code> and store the model version beside the vector, so re-running a failed batch never double-charges or duplicates vectors.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Two facts shape the design more than any tuning. Embedding is typically one to two orders of magnitude cheaper per token than generation, so ingestion cost is driven by <em>tokens \u00d7 volume<\/em> \u2014 which makes deduplication and boilerplate stripping, both of which run before the embedding call, your highest-leverage optimizations. And a model change is a <strong>migration, not a sync<\/strong>: vectors from two models cannot share an index, so you build a second index, backfill through the batch path, shadow-read to compare quality, then cut over. Because you kept raw artifacts and a parse-config version, that is a re-index, not a re-crawl.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Running the hot path and the backfill through one OpenAI-compatible endpoint removes a class of provider plumbing \u2014 <a href=\"https:\/\/qoraapi.com\/\" target=\"_blank\" rel=\"noopener\">qoraapi.com<\/a> exposes embeddings and chat models behind a single API, which makes swapping the embedding model a config change rather than an integration project. For how vectors and retrieval fit together, start with <a href=\"https:\/\/qoraapi.com\/blog\/ai-embeddings-rag\/\">embeddings and RAG<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">How often should the ingestion sync run?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Split by urgency, not one cron interval. Run incremental updates every 5\u201315 minutes where staleness is visible to users, and hourly or daily for slow-moving corpora. Run full-ID reconciliation on a slower cadence \u2014 daily for high-churn sources, weekly for stable ones \u2014 because it is the only job that catches documents deleted while a connector was down.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Do I need to re-embed when only metadata changes?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">No. Separate the content hash from the metadata hash. If the text is byte-identical after normalization the vector is still valid \u2014 update metadata in place and skip the embedding call. This matters most for ACL changes, which are frequent and should never cost an embedding pass.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is the minimum viable ingestion pipeline?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Five components: a connector that records a version identifier per document, a parser with a validation gate that quarantines low-yield output, a chunker storing small children plus large parents, an upsert keyed on a content hash, and a tombstone-aware delete path. Ship that before adding reranking or hybrid search.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What should I do with documents the parser fails on?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Quarantine them, do not index them. Route low-yield documents to a review queue, notify the source owner, and keep the raw artifact so a parser improvement can reprocess them. An empty or truncated document is worse than a missing one: retrieval will happily return it and the model will answer from it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">RAG quality is decided before retrieval runs. Parse with layout awareness and fail loudly on low-yield documents. Keep heading structure so chunking follows document boundaries instead of arbitrary token windows, embed small children while storing large parents, and prefix every chunk with its heading path. Sync incrementally with content hashes, reconcile deletions with tombstones, and propagate source ACLs onto every chunk so the vector store filters before it ranks. Then make the backfill boring: batch it, cap concurrency, keep jobs idempotent.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Get those right and retrieval becomes an optimization problem instead of a debugging exercise. Get them wrong and no amount of reranking will save you.<\/p>\n\n\n\n\n<h3 class=\"wp-block-heading\">Related reading<\/h3>\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/qoraapi.com\/blog\/production-rag-architecture\/\">Building Production RAG: Chunking, Hybrid Search, and Re-Ranking<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-embeddings-rag\/\">AI Embeddings Explained: Vectors, Similarity, and Building Your First RAG<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/batch-ai-api-processing\/\">Batch AI APIs: Processing Millions of Requests Affordably<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/document-data-extraction\/\">Extracting Structured Data from Documents with AI APIs<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-copilot-in-app\/\">Building an In-App AI Copilot: Architecture, UX, and Guardrails<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/detect-reduce-hallucinations\/\">Detecting and Reducing Hallucinations in Production LLM Apps<\/a><\/li><li><a href=\"https:\/\/qoraapi.com\/blog\/ai-gateway-vs-api-gateway\/\">AI Gateway vs API Gateway: Key Differences and When to Use Each<\/a><\/li><\/ul>\n\n","protected":false},"excerpt":{"rendered":"<p>Most RAG failures start at ingestion. Build a pipeline that parses documents, chunks for retrieval, syncs incrementally with tombstones, and propagates permissions.<\/p>\n","protected":false},"author":1,"featured_media":161,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[5,6,9,7],"class_list":["post-162","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-api","tag-ai-api","tag-api-gateway","tag-developer-tools","tag-developers"],"_links":{"self":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/162","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/comments?post=162"}],"version-history":[{"count":2,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/162\/revisions"}],"predecessor-version":[{"id":269,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/posts\/162\/revisions\/269"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media\/161"}],"wp:attachment":[{"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/media?parent=162"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/categories?post=162"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qoraapi.com\/blog\/wp-json\/wp\/v2\/tags?post=162"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}