The thesis
Retrieval is engineering, not magic. The vendor that sells you "vector search" sells you one third of a system. The team that builds against that one third and wonders why their RAG hallucinates is rediscovering what the search-engineering community has known for two decades: lexical and semantic and structural retrieval each find a different kind of relevance, and a production retrieval stack composes all three. There are tools that make this composable. There are patterns that make it cheap. None of it is exotic. All of it is what separates a search demo from a search system.
This article is the practitioner companion to the broader RAG Knowledge Engines piece. RAG Engines covers the full architecture (chunker, index, citations, evals, when not to RAG). This piece zooms in on the retrieval layer specifically: which tools to use, how to compose them, how to handle queries before they hit the index, and how to keep the whole thing fast enough to be interactive.
The three retrieval stacks
Lexical search (BM25) is the Okapi-derived ranking function that scores documents based on term frequency, inverse document frequency, and document length. It is forty years old and still the right default for exact-phrase and named-entity queries. Typesense and OpenSearch are the production-grade implementations I reach for. Both have native typo tolerance, faceting, and per-field weighting. Both run on commodity hardware at scale. The configuration cost is hours, not weeks.
Dense vector search (cosine similarity over learned embeddings) is the modern technique that gets all the press. The query and every document chunk are embedded into a high-dimensional vector space (Gemini Embedding 2 at 3072 dimensions is my default; OpenAI's text-embedding-3-large is the comparable alternative). Similarity in that space approximates semantic similarity. Qdrant is the production-grade vector store I prefer; pgvector if the team is already on Postgres and the corpus is under a million chunks. Both ship HNSW indexes, filter-aware search, and reasonable defaults.
Graph retrieval traverses typed relationships between entities to find structurally relevant context the other two cannot see. Neo4j with Graphiti is my default for the temporal-knowledge-graph case (institutional memory across time, with edge-level invalidation when facts change). For lighter cases, a simpler graph store (or even a Convex table modeling edges directly) is enough. The graph is not for every retrieval; it is for the queries where structure matters more than text similarity.
How the three compose
The naive composition is "run all three, union the results." That works for small corpora and breaks at scale because the union has too many candidates and the model gets noisy context. The grown-up composition is "run all three with bounded K per source, union with deduplication, rerank the union with a cross-encoder, take top N for the prompt."
The cross-encoder reranker is the load-bearing piece most teams skip. It takes each (query, candidate) pair and produces a relevance score in a single forward pass. Cohere Rerank is the easiest hosted option. The bge-reranker family from BAAI is the strongest open-weights option; running it on GPU is straightforward and the latency is manageable for top-50-into-top-10 reduction. The key property is that cross-encoders see both query and candidate together, so they catch contextual relevance that bi-encoder embeddings (which see them separately) cannot.
The numbers that matter: BM25 returns top-30, dense returns top-30, graph returns top-10 (graph queries are inherently more selective). Union and dedupe down to roughly forty candidates. Reranker scores all forty. Take top eight or ten for the prompt. Latency budget for the full retrieval-and-rerank pipeline runs about 200-400ms on a warm cache, which is fast enough to feel interactive in a chat UI.
Query rewriting and semantic routing
The user's query is rarely the right query for the index. Users underspecify. They use pronouns referring to prior turns. They mix multiple questions into one. They use brand-new terminology the index does not contain. The retrieval layer has to handle this translation gap, and the translation is itself a model task.
Query rewriting is the pattern of running the user query through a small model that emits one or more rewritten queries optimized for retrieval. The simplest version expands "what's the cap?" to "indemnification cap MSA-2024-014 schedule 2." A more capable version emits multiple variant queries to run in parallel, broadening recall. Each rewritten query gets retrieved against, the union goes to the reranker, and the reranker handles the deduplication. The cost is one extra small-model call per user query, well worth it for the recall lift on the kinds of queries production users actually ask.
Semantic routing is the upstream sibling: the system decides which retrieval stack (or which subset of the corpus) to use based on the query shape. A query for a specific clause routes to the lexical stack with high weight on the contracts collection. A query asking for a synthesis routes to the dense stack across the full corpus. A query about relationships between entities routes to the graph. The router is a small classifier (a fine-tuned model or a structured-output prompt) that takes the query and emits a route plus configuration. The overhead is one classification call. The savings is not running all three stacks against the full corpus when only one is relevant.
Retrieval-augmented evals
Eval discipline for retrieval is the same shape as eval discipline anywhere: a held-out set of queries with known-correct chunks, a runner that executes the stack and checks recall@K and rerank position of the correct chunk, a CI integration that runs on every change. The retrieval-specific twist is that the eval has to test both the retrieval (did the right chunk get into the candidate set) and the rerank (did the right chunk end up in the top N).
Two metrics matter most. Recall@30 measures whether the candidate set included the right chunk before reranking; this catches retrieval-stack failures. Mean reciprocal rank of the correct chunk in the reranked top-10 measures whether the reranker placed the right chunk near the top; this catches reranker failures. When recall@30 drops, the chunker or the index broke. When MRR drops but recall@30 holds, the reranker broke or the prompt changed in a way that affected reranking. These are different fixes for different failures, and the metrics tell you which is which.
The other gauge is production. Log every retrieval as a LogFire span with the query, the per-stack candidate sets with scores, the reranked output, the chunks that landed in the prompt, and any user feedback signals. The dashboard surfaces queries with low rerank scores (the system was not confident), queries with high reranker disagreement (the candidate set was diverse), and queries with thumbs-down feedback (the user disagreed with the answer). These three views answer "where is retrieval failing in production" without requiring the operator to read raw logs.
What this gets you
A retrieval layer built this way is the substrate every other AI feature in the system depends on. The RAG agent gets better answers. The semantic search UI returns more relevant results. The content-compiler's evidence-block resolution finds the right citations. The agent's tool calls land on the right context. The investment compounds across every downstream feature because retrieval is a horizontal capability, not a vertical one.
If you are evaluating a retrieval team or vendor, the questions are about which stacks they compose, whether they ship a reranker, whether they handle query rewriting and semantic routing, whether they have a recall-and-MRR eval harness, and whether their production observability surfaces low-confidence retrievals. If they sell you "vector search" and stop there, what they have is one third. If they walk you through composition and rerank and rewriting and evals, what they have is the system.