Building a RAG Ingestion Pipeline: Crawling, Parsing, and Syncing

A RAG ingestion pipeline is an ETL job in three stages: crawl sources into raw documents, parse and normalize them into clean text plus metadata, then chunk, embed, and sync into a vector index — incrementally, using content hashes for updates and tombstones for deletes. Retrieval quality is permanently capped by what this pipeline emits.

Everything downstream operates on the text your parser produced. This guide covers the ingest side: connectors, parsing, chunk storage, incremental sync, ACL propagation, and embedding economics.

Why ingestion is where RAG projects quietly fail

The symptom is always the same. A demo works on five hand-picked PDFs; three weeks later users say the assistant is confidently wrong, and the team reranks, swaps embedding models, and tunes top_k. Nothing fixes it, because the defect is upstream: a scanned contract parsed to an empty string, a wiki sidebar repeated in every chunk, a deleted policy still being cited.

The economics are asymmetric. A retrieval bug affects one query. An ingestion bug affects every query that touches that document, forever — and it stays invisible in your metrics, because retrieval is working correctly: it faithfully returns the garbage you stored.

So treat parsing as a validated step with an explicit contract. Two rules carry most of the weight:

• Measure parse yield. Compute tokens-per-page for every document and set a floor — say 200 tokens per page for prose PDFs. Anything below it goes to quarantine for human review, never into the index.
• Fail loudly, never silently. A parser returning 300 tokens from a 40-page report has “succeeded” and poisoned your index. Assert on page count, heading count, and table count; fail the document when the ratio collapses.

Keep the raw artifacts. An index is a build artifact — you should be able to rebuild it from raw documents plus a parse-config version. Teams that discard originals can never fix a parser bug retroactively or change embedding models cheaply.

Sources and connectors: what each one actually needs

A connector is not “download the file.” Each source class carries different metadata, deletion semantics, and failure modes:

Source	What you get	What the connector must handle
Docs in Git (Markdown / MDX)	Text + frontmatter	Read from the repo, not the rendered site — you keep history, frontmatter, and a commit SHA to version chunks with. Deletions arrive as a git diff.
Wikis (Confluence, Notion)	Block JSON or HTML	Pagination, nested child pages, per-page permission lists, and an updated_at that actually changes. Strip editor chrome.
PDFs (contracts, specs, scans)	Binary; may lack a text layer	Layout-aware parse with reading-order reconstruction, OCR fallback for image-only pages, page numbers preserved for citations.
Databases	Rows	Incremental by an updated_at/id watermark. One text projection per row — never dump whole tables. Mirror row permissions into a groups column at ingest.
SaaS APIs (tickets, issues)	Nested JSON	Separate description from comment thread, filter closed/resolved noise if users only search live work, redact PII before embedding.
Public web pages	HTML with chrome	Respect robots.txt and crawl rate, extract the main content region only, record fetch time — web pages have no updated_at.

The governing rule: prefer the source of truth that carries the metadata you need. Rendering a docs site to HTML throws away git history and ACLs; pulling the same content through the repository API keeps both.

Parsing: layout, structure, and tables

Text extraction is solved only for plain text. Real corpora pose three problems at once.

Reading order is where naive PDF extraction dies. Multi-column layouts, sidebars, and footers get interleaved into nonsense. A layout-aware extractor reconstructs blocks by position and drops repeated header/footer bands. Check for a text layer first: a page yielding fewer than ~50 characters is almost certainly an image needing OCR — for that page only.

Structure preservation means keeping the heading hierarchy in the extracted text, because you need it for chunking and cannot rebuild it later. Convert headings to a marked form (#, ##) at parse time so the chunker splits on document structure, not a fixed token count.

Tables must be emitted as tables — Markdown or HTML — never flattened prose. Flattening destroys the row-column binding, so Q1 | 12% | 8% becomes an unattributable string of numbers. When a table spans chunks, prepend the header row to every slice.

Here is a working parse-and-normalize step with validation built in:

import hashlib, re
from dataclasses import dataclass, field
from bs4 import BeautifulSoup
import fitz  # PyMuPDF

MIN_CHARS_PER_PAGE = 50
DROP = {"script", "style", "nav", "footer", "aside", "form", "noscript"}
HEADING = re.compile(r"^h[1-6]$")

@dataclass
class Doc:
    doc_id: str
    source: str
    uri: str
    text: str
    metadata: dict = field(default_factory=dict)
    content_hash: str = ""

def normalize(text: str) -> str:
    text = text.replace("\u00ad", "")             # soft hyphens
    text = re.sub(r"[ \t]+", " ", text)
    text = re.sub(r"\n{3,}", "\n\n", text)
    text = re.sub(r"(?m)^\s*\d+\s*$", "", text)   # bare page numbers
    return text.strip()

def parse_pdf(path: str) -> Doc:
    pdf, pages, ocr_pages = fitz.open(path), [], 0
    for i, page in enumerate(pdf):
        raw = page.get_text("text")
        if len(raw.strip()) < MIN_CHARS_PER_PAGE:
            ocr_pages += 1
            raw = ocr_page(page)                  # your OCR adapter
        pages.append(f"[page {i+1}]\n{normalize(raw)}")
    # fail loudly: an image-only PDF whose OCR we cannot vouch for
    if ocr_pages > len(pages) * 0.8:
        raise ValueError(f"{path}: image-only, OCR quality unverified")
    return Doc(doc_id=path, source="pdf", uri=path,
               text="\n\n".join(pages),
               metadata={"pages": len(pages), "ocr_pages": ocr_pages})

def parse_html(html: str, uri: str) -> Doc:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.find_all(DROP):
        tag.decompose()
    root = soup.find("main") or soup.find("article") or soup.body or soup
    # keep structure: mark headings so the chunker can split on them
    for h in root.find_all(HEADING):
        h.insert_before(f"\n\n{'#' * int(h.name[1])} ")
    return Doc(doc_id=uri, source="html", uri=uri,
               text=normalize(root.get_text("\n")),
               metadata={"title": (soup.title.string or "").strip()})

def with_hash(doc: Doc, parse_cfg: str = "v3") -> Doc:
    # hash NORMALIZED text + parse config, not raw bytes: a re-exported
    # PDF that only changes timestamps must not trigger re-embedding.
    payload = f"{parse_cfg}\n{doc.text}"
    doc.content_hash = hashlib.sha256(payload.encode()).hexdigest()
    return doc

Two details matter most. The hash covers normalized text plus a parse_cfg version, so a parser upgrade deliberately invalidates everything while a no-op file change does not. And the OCR guard raises instead of returning a plausible stub — preventing the most common silent failure in document RAG.

Chunking at ingest vs at query time

Most chunking advice conflates two decisions. Separate them and the design becomes obvious.

• Ingest-time chunking decides what you embed and what you store — not the same unit. Embed small children (200–500 tokens) so the vector matches a short query precisely; store the larger parent section (1,000–2,000 tokens) keyed by parent_id.
• Query-time assembly decides what the model sees: retrieve the children, then expand each hit to its parent — “small-to-big”. Precise matching and full context in one request, without guessing a single chunk size that satisfies both.

Two techniques belong to the ingest side and cost nothing at query time:

• Contextual prefixes. Prepend the document title and heading path to each child before embedding: Billing API > Rate limits > Burst allowance. A 200-token chunk is often ambiguous alone; the breadcrumb stops it colliding with every other “…allowance” paragraph in the corpus.
• Structural boundaries over fixed windows. Split on headings, list items, and table rows first; fall back to a token window only when one section exceeds the limit. A fixed 512-token window slices tables in half and splits procedures between steps 4 and 5. Overlap of 10–15% at that boundary is a sane default.

Everything on the retrieval side — hybrid search, reranking, fusion, context budgeting — belongs to our guide on production RAG. Ingestion owns text quality, chunk identity, and metadata; retrieval owns ranking and assembly.

Incremental sync and change detection

Full re-ingestion is fine at 5,000 chunks and ruinous at 5 million. Incremental sync classifies each document into a change type and takes the cheapest correct action:

Change type	Signal	Action
New document	doc_id absent from the index	Parse → chunk → embed → upsert
Content updated	content_hash differs	Re-parse; re-embed only chunks whose own hash changed; delete orphaned chunk ids
Metadata-only change (title, ACL)	metadata_hash differs, content_hash unchanged	Update metadata in place — no re-embedding
Deleted at source	Absent from a full source listing, or deleted_at set	Tombstone: mark deleted, remove vectors, exclude from search
Moved or renamed	Same content_hash, new URI	Update uri and metadata only
Source unreachable	Connector error or timeout	Do nothing — never tombstone on a fetch failure

The row most pipelines get wrong is deletion. Upsert-only ingestion means deletes never propagate, so deprecated policies and removed customer data stay retrievable — and get cited with full confidence. Two detection strategies trade off differently:

• CDC / deleted_at: cheap and near-real-time, but only if the source exposes deletions. Many do not.
• Full-ID reconciliation: list every source id, diff against the index, tombstone the difference. Expensive but authoritative — the only approach that catches documents deleted while your connector was down.

Run reconciliation on a cadence — daily for high-churn sources, weekly for stable ones. Implement tombstones as soft deletes: set deleted: true plus deleted_at, filter them at query time, purge after a retention window. That makes a bad connector run reversible instead of catastrophic.

Watermark sync has two traps. Subtract a safety lag (5–15 minutes) from the watermark, because transactions commit out of order and clocks skew; without it you silently skip rows. And use a composite (updated_at, id) cursor rather than a timestamp alone, or rows sharing a timestamp get skipped on ties. Commit the watermark only after the write succeeds:

def sync_table(conn, index, cursor):
    rows = conn.execute(
        """select id, body, updated_at from docs
           where (updated_at, id) > (%s, %s)
             and updated_at < now() - interval '10 minutes'
           order by updated_at, id limit 500""",
        (cursor["updated_at"], cursor["id"])).fetchall()

    for row in rows:
        doc = with_hash(parse_row(row))          # content_hash + metadata_hash
        if index.get_hash(doc.doc_id) == doc.content_hash:
            continue                             # no-op: costs zero embeddings
        index.upsert(embed_chunks(doc))          # only changed chunks embed

    if rows:
        index.commit()
        cursor.update(updated_at=rows[-1].updated_at, id=rows[-1].id)
    return len(rows)

def reconcile_deletes(conn, index):
    live = {r.id for r in conn.execute("select id from docs")}
    stale = index.list_doc_ids() - live
    index.tombstone(stale)                       # soft delete, purge later
    return len(stale)

Note what the watermark cannot do: it cannot see deletions. Watermarks handle updates; reconciliation handles deletes. You need both.

Metadata and permissions: an index without ACLs leaks data

A vector store has no row-level security by default. The moment you ingest an HR policy, a private ticket, or a restricted repository, you have created a shadow copy of your most sensitive content behind a single API key. This is the highest-severity failure mode in the pipeline, and it is entirely an ingestion problem.

The pattern that works: denormalize ACLs at ingest, filter at query time, and let the vector store enforce it.

• Store permission fields on every chunk — acl_groups: ["eng", "sre"] — copied from the parent at ingest. Denormalized, because filtering happens per chunk.
• Pass the filter into the ANN search itself, built from the caller’s verified identity: filter={"acl_groups": {"$in": user_groups}}. The store never considers vectors the caller cannot see.
• Never retrieve-then-filter. Fetching top_k=10 and dropping 8 unauthorized hits gives a worse answer, wastes tokens, and turns one missing filter into a data breach.
• Treat ACL changes as content changes — the metadata-only row in the sync table. Update in place, skip the embedding call.
• Multi-tenant deployments need a per-tenant namespace and a per-tenant filter. The namespace bounds blast radius and query cost; the filter is the security control.

Where a database’s permission model cannot be expressed as groups, resolve it at ingest with a join that materializes a groups column per row. Evaluating a live model at query time puts a database round-trip inside your search path and asks the vector store to enforce authorization it cannot see. Record the ACL snapshot version on every chunk so you can answer “why did this user see that document in March.”

Cost and rate limits at scale

Backfilling 500,000 chunks at roughly 400 tokens each is about 200 million tokens of embedding work. Sent synchronously at a few dozen requests per second, that is days of wall clock and a permanent stream of 429s. The methodology that keeps it manageable:

• Batch the payload. Embedding endpoints accept arrays, so send many chunks per request. For an initial backfill or re-embed, route it through batch AI APIs where latency does not matter — the discount is substantial and throughput far higher.
• Split hot and cold paths. Newly changed documents must be searchable in minutes and go through a small synchronous pool; backfills and model migrations go through the batch path. One queue for both means a backfill starves your freshness SLA.
• Cap concurrency, then back off. Start at 4–8 in-flight requests, add exponential backoff with jitter on 429, and honor Retry-After — see our rate-limit handling guide. A retry must never re-embed chunks that already committed.
• Make jobs idempotent. Key every job by (chunk_id, embed_model, parse_cfg) and store the model version beside the vector, so re-running a failed batch never double-charges or duplicates vectors.

Two facts shape the design more than any tuning. Embedding is typically one to two orders of magnitude cheaper per token than generation, so ingestion cost is driven by tokens × volume — which makes deduplication and boilerplate stripping, both of which run before the embedding call, your highest-leverage optimizations. And a model change is a migration, not a sync: vectors from two models cannot share an index, so you build a second index, backfill through the batch path, shadow-read to compare quality, then cut over. Because you kept raw artifacts and a parse-config version, that is a re-index, not a re-crawl.

Running the hot path and the backfill through one OpenAI-compatible endpoint removes a class of provider plumbing — qoraapi.com exposes embeddings and chat models behind a single API, which makes swapping the embedding model a config change rather than an integration project. For how vectors and retrieval fit together, start with embeddings and RAG.

Frequently asked questions

How often should the ingestion sync run?

Split by urgency, not one cron interval. Run incremental updates every 5–15 minutes where staleness is visible to users, and hourly or daily for slow-moving corpora. Run full-ID reconciliation on a slower cadence — daily for high-churn sources, weekly for stable ones — because it is the only job that catches documents deleted while a connector was down.

Do I need to re-embed when only metadata changes?

No. Separate the content hash from the metadata hash. If the text is byte-identical after normalization the vector is still valid — update metadata in place and skip the embedding call. This matters most for ACL changes, which are frequent and should never cost an embedding pass.

What is the minimum viable ingestion pipeline?

Five components: a connector that records a version identifier per document, a parser with a validation gate that quarantines low-yield output, a chunker storing small children plus large parents, an upsert keyed on a content hash, and a tombstone-aware delete path. Ship that before adding reranking or hybrid search.

What should I do with documents the parser fails on?

Quarantine them, do not index them. Route low-yield documents to a review queue, notify the source owner, and keep the raw artifact so a parser improvement can reprocess them. An empty or truncated document is worse than a missing one: retrieval will happily return it and the model will answer from it.

Conclusion

RAG quality is decided before retrieval runs. Parse with layout awareness and fail loudly on low-yield documents. Keep heading structure so chunking follows document boundaries instead of arbitrary token windows, embed small children while storing large parents, and prefix every chunk with its heading path. Sync incrementally with content hashes, reconcile deletions with tombstones, and propagate source ACLs onto every chunk so the vector store filters before it ranks. Then make the backfill boring: batch it, cap concurrency, keep jobs idempotent.

Get those right and retrieval becomes an optimization problem instead of a debugging exercise. Get them wrong and no amount of reranking will save you.

Related reading

• Building Production RAG: Chunking, Hybrid Search, and Re-Ranking
• AI Embeddings Explained: Vectors, Similarity, and Building Your First RAG
• Batch AI APIs: Processing Millions of Requests Affordably
• Extracting Structured Data from Documents with AI APIs
• Building an In-App AI Copilot: Architecture, UX, and Guardrails
• Detecting and Reducing Hallucinations in Production LLM Apps
• AI Gateway vs API Gateway: Key Differences and When to Use Each