Better Embeddings Won’t Fix Missing Provenance in RAG

The author built a four‑stage retrieval‑augmented generation system that ingests roughly 5,000 internal documents—including PDFs with tables and three versions of a parental‑leave policy spanning five years—by extracting raw text, chunking it into 512‑token windows with 50‑token overlap, embedding each chunk with OpenAI’s text‑embedding‑3‑small (1536‑dimensional vectors), and persisting the vectors in a PostgreSQL table via the pgvector extension. Retrieval selects the five nearest chunks by cosine similarity, feeds them to GPT‑4o‑mini, and the LLM is instructed to answer only from the supplied context. When a user asked about parental‑leave entitlement, the system cited a 2021 policy document that had since been superseded, because the vector store contained only an id, the chunk text, and the embedding—no document identifier, revision date, or source path. The bug surfaced only after three weeks of unnoticed incorrect answers, illustrating that flawless embeddings can still retrieve the wrong version when provenance data is absent.

This failure mirrors a broader pattern in the rapidly expanding RAG market, where startups and cloud providers ship turnkey pipelines that prioritize vector similarity over metadata integrity. Tools such as LangChain, LlamaIndex, and commercial vector databases (e.g., Pinecone, Weaviate) often expose simple “store‑and‑search” APIs, encouraging engineers to drop in default chunking and embedding settings without considering versioning or hierarchical document structures. As enterprises adopt RAG for compliance‑sensitive domains—HR policies, financial regulations, medical guidelines—the lack of built‑in provenance becomes a liability, potentially leading to regulatory breaches or costly misinformation. The article’s “naïve baseline” is representative of many production deployments that overlook the need for hybrid search (keyword + vector) and explicit metadata columns.

Going forward, RAG architects must enrich vector tables with document IDs, timestamps, and status flags, and couple similarity search with deterministic filters that enforce recency or authoritativeness. Automated changelog ingestion, hierarchical chunking that preserves section headings, and reranking stages that weigh provenance will be essential to prevent stale citations. The series promises to address these gaps in subsequent posts, but the immediate takeaway is that without systematic provenance tracking, even state‑of‑the‑art embeddings cannot guarantee accurate, up‑to‑date answers.