The thesis
A retrieval-augmented generation system is an answer machine bolted to a search engine. The interesting failure mode is not the model. It is the search. When a RAG demo collapses on a real corpus, the model is almost never the cause. The retrieval layer pulled the wrong chunks, the chunks were the wrong shape, the chunks lost the document structure they came from, or the answer was rendered without any auditable link back to where it came from. Fix the retrieval and ninety percent of the hallucination story disappears with it.
The frame I work from is simple. Embeddings are necessary and not sufficient. Semantic search finds nearby meaning; it does not find structurally relevant context. Graphs find structurally relevant context; they do not find nearby meaning. The hybrid is not optional once the corpus is real. And the answer that gets returned to the user has to carry its sources with it as first-class objects, not as a post-hoc footnote. If the citation is a string concatenation, the system is not auditable. If the citation is a typed reference back to a chunk that knows its parent document and its parent section, the system can be debugged the way you debug any other production data path.
Why vector-only fails on a real corpus
The vector demo wins because the demo corpus is small and the questions are flattering. Twenty PDFs. A handful of curated queries. The cosine similarity finds plausible chunks and the model writes something coherent. That looks like working. It is not.
Three things break the moment the corpus crosses a few thousand documents. The first is recall. Dense embeddings cluster meaning, not exact phrases. When a user asks for a clause that contains a specific defined term ("Material Adverse Event," "Permitted Transferee," "Net Revenue Retention") the embedding does not know that term is load-bearing. It finds chunks that talk around the topic and misses the chunk where the term is actually defined. BM25 catches this trivially because lexical match is what BM25 does. Hybrid stacks exist because each half compensates for the other half's blind spot.
The second is structure loss. A large corpus is not a pile of independent essays. It is a hierarchy. Contracts are the clearest example. Master agreement points to schedules. Schedules reference exhibits. Exhibits reference defined terms in the master. A naive chunker reading those documents into 800-token windows shreds the hierarchy and the embedding has no way to reconstruct it. The user asks "what is the indemnification cap" and the system returns a chunk from the schedule that quotes the cap, missing the master clause that says the schedule's cap is overridden by the side letter. The answer is wrong by structural omission, not by retrieval distance.
The third is freshness. Embedding indexes go stale. New documents land. Old ones get superseded. The naive build re-embeds the full corpus on every change. The grown-up build maintains an incremental index keyed on document version with a backfill path for re-embedding when the underlying model changes. This is not exotic. It is the difference between a system that runs and a system that quietly drifts until the day a user catches it citing a contract that was rescinded six months ago.
The hybrid architecture in practice
The shape that works is three retrieval stacks composed behind a single retrieval-and-rerank API. Lexical search (BM25 via Typesense or Elasticsearch) catches exact phrases and named entities. Dense vector search (Qdrant, pgvector, or whatever you are running) catches semantic neighborhoods. Graph traversal (Neo4j with Graphiti, or a lighter-weight graph store) catches structural relationships the other two cannot see. Each returns a candidate set with scores. A reranker, usually a cross-encoder (Cohere Rerank, or a fine-tuned bge-reranker), takes the union, evaluates each candidate against the query in pairwise fashion, and produces a final ordered list. The model never sees the raw retrievals. The model sees the top N reranked chunks plus their metadata.
The metadata is where the system earns its keep. Every chunk carries a document ID, a section path (an array of section IDs from root to leaf), a span (start and end character offsets in the source), a version, and a checksum. The model is instructed to answer the question and to attach a citation array where each citation is a typed reference to one or more of those chunks. The rendering layer turns the citation array into footnotes the user can click. If the user clicks a citation, the rendering layer fetches the chunk by ID, fetches the parent document, and shows the user the exact span highlighted in context. If the chunk version no longer matches the live document version, the rendering layer warns the user and offers to re-resolve.
This sounds like a lot. It is a lot. But every piece is load-bearing. Without lexical, the named-entity question fails. Without dense, the conceptual question fails. Without graph, the structural question fails. Without rerank, the union is too noisy for the model. Without typed citations, the system is unauditable. Skipping any of the five components saves a week of build time and costs you the production deployment.
Citations as first-class objects
The phrase "citations as first-class objects" is a load-bearing one. A first-class citation is a typed Pydantic model that lives in the database, has a UUID, references a chunk and a document and a version, and is queryable. A footnote in a string is a presentational artifact. The presentational artifact is the last step of rendering, not the data shape.
The reason this matters is debugging. When a user reports a wrong answer, the question is not "why did the model say that." The question is "which chunks did the retrieval return, what was the rerank score, what citations did the model attach, and which of those citations actually supported the claim the user is disputing." If the citations are strings, the answer is reconstructive guesswork. If the citations are typed objects with foreign keys, the answer is one query: pull the citation, pull the chunk, compare to the model's claim, identify whether the failure was retrieval, rerank, or generation. That single query saves entire days of debugging on a production system.
The downstream affordance is also material. Users trust answers that show their work. The citation pill that opens an inline panel showing the source paragraph with the relevant span highlighted converts a doubtful user into an evaluating user. The doubtful user closes the tab. The evaluating user comes back with a follow-up question. The system that supports evaluation grows the relationship; the system that does not, does not.
How you know retrieval is working
Every RAG system needs an eval harness and most do not have one. The harness is a set of held-out questions with known-correct chunks. For each question, you run the retrieval stack, you check whether the top K results contain the known-correct chunk, and you log the recall@K, the rerank position of the correct chunk, and the latency. You do this on every deploy. You do this on every embedding model upgrade. You do this when you change the chunker.
The eval set is not glamorous. It is a CSV. The first time you build one, you sit with a domain expert for half a day and write thirty questions. Thirty is enough to catch most regressions. Three hundred is enough to claim statistical significance on small differences between retrieval configurations. The number you pick is determined by how often you intend to change the retrieval stack and how much you care about the difference between configurations. Most teams do not need three hundred. Most teams need thirty and the discipline to actually run them on every change.
The other gauge is production. Log every retrieval. Log the query, the candidate set with scores, the reranked top N, the chunks that ended up in the prompt, the citations the model produced, and the user's downstream behavior (did they click a citation, did they ask a follow-up, did they give a thumbs-down). LogFire spans against this data answer the question "what is going wrong, where, and how often" in a single dashboard. Without those spans you are running blind on a system whose failures are silent until they are catastrophic.
When RAG is the wrong tool
RAG is the right answer when the corpus is large, the queries are open-ended, and the user benefits from being shown the source. RAG is the wrong answer when any of those three conditions is missing. If the corpus is small (under a few hundred pages of substance), put the whole thing in the context window and stop pretending you need retrieval. If the queries are narrow (always one of five forms), build a structured pipeline with explicit lookups and let the model do final synthesis on a known set of fields. If the user does not benefit from sources (a chat interface for casual questions), citations are noise and a smaller, faster, cheaper architecture wins.
The pattern I see most often in client work is RAG used because it is the visible solution, when the actual problem is a structured-data problem dressed up in document language. The fix in those cases is not better retrieval. It is correctly identifying that the system needs schemas and mappings and a SQL or Convex query, with the model on top doing language framing rather than language reasoning. RAG is a hammer; not every nail.
The metagraph
The graph half of the hybrid stack hints at something larger. Treat the relationships between concepts as first-class and retrieval changes. It stops searching for the nearest chunks and starts navigating how the corpus actually hangs together. I think about a body of knowledge the way a bat thinks about a cave. You don't get the room by shining a flashlight at one wall. You send signals into the whole space and rebuild it from what comes back. Map the concepts, map how they connect, go several layers deep, and what you end up with is a world model you can query.
This is the metagraph. Knowledge held as structure rather than as a pile of passages that happen to sit near each other. Two paragraphs can be neighbors in embedding space and mean opposite things, because one was written by an authority and one by a stranger, one is current and one was superseded last quarter, one states a claim and one refutes it. Flat retrieval averages all of that into a single fuzzy neighborhood and loses the distinctions that decide the answer. A metagraph keeps them. Who said it, when it was true, what it contradicts, and where it came from each become a property on the graph itself. Retrieval then walks that structure. It follows a claim to its source, a term to its definition, a decision to the three decisions that depended on it. The multi-hop questions flat search can't reach turn into a traversal.
The same frame works on a market. A customer lives inside an economy, and that economy has its own customers, its own pressures, its own history. Model that as a graph and you can navigate the relationships between concepts in ways a keyword match or a cosine score never surfaces. The full treatment lives in the WikiDesignCo library. From RAG to Metagraph picks up exactly where this article stops, on the architecture. Echolocation is the long read on the sensing method itself, reading a market by what pings back instead of by its demographics.
What this gets you
A RAG system built this way does the unglamorous thing of staying right. My own retrieval engine indexes a 5-million-word personal corpus (hundreds of Markdown files, PDF textbooks, and YouTube transcripts) on the open-source Archon platform, with document-type-specific chunkers and task-typed Gemini embeddings. The citation accuracy climbing past baseline RAG is not a model achievement. It is a retrieval-architecture achievement. The base model stayed the same off-the-shelf model. What changed was the chunker (structure-aware, per document type), the index (hybrid), the rerank (cross-encoder), and the citation surface (typed, queryable, clickable). Every one of those changes was boring. The compounding of all of them is what made retrieval trustworthy.
If you are evaluating a RAG vendor or an internal team, the questions to ask are not about the model. The questions are about the chunker, the index topology, the rerank strategy, the citation data shape, the eval harness, and the production observability. If they cannot answer those questions with specifics, what they have built is a demo. If they can, what they have built is a system.