Dense, Sparse & Hybrid Retrieval

Dense retrieval embeds query and chunks into vectors and ranks by cosine similarity (Embeddings). It matches meaning: "can't log in" finds "authentication failure troubleshooting". It is weak on tokens it has never seen or that carry meaning by exact form — order numbers, error codes, function names, version strings, people's names.

Sparse retrieval — classically BM25 — represents text as a bag of terms and scores a chunk by how many query terms it contains, weighted by how rare each term is across the corpus (IDF) and dampened by chunk length. It is exact and explainable: E4021 matches only chunks containing E4021. It is blind to synonyms and paraphrase.

These weaknesses are complementary, which is why production systems rarely run one alone. The question is not "dense or sparse" but "how do I combine them and when does each dominate".

Consider a support corpus and the query "what does error E4021 mean". The dense model sees "error … mean" and returns generic troubleshooting chunks; the digits contribute little to the vector, and E4012 embeds almost identically to E4021. BM25 sees a rare token e4021 with high IDF and returns exactly the chunk that defines it. Any corpus heavy in IDs, SKUs, ticket numbers, code symbols, or legal clause numbers has this shape.

The reverse case: "how do I get my money back" against a document that only ever says "refund". BM25 finds nothing — no shared terms. Dense retrieval matches trivially. A user population that mixes both query styles (which is every user population) needs both retrievers.

You cannot add a cosine score of 0.82 to a BM25 score of 14.3 — they are on unrelated scales. Reciprocal Rank Fusion (RRF) sidesteps this by combining *ranks* instead of scores: each chunk gets Σ 1 / (k + rank_i) over the lists it appears in, with k commonly 60. A chunk ranked well by both retrievers rises to the top; a chunk found by only one still gets credit.

Worked example with k = 60. Dense returns [A, B, C]; BM25 returns [C, D, A]. Scores: A = 1/61 + 1/63 ≈ 0.0164 + 0.0159 = 0.0323. C = 1/63 + 1/61 = 0.0323 (tie with A; break by original best rank or by a tiny weight). B = 1/62 ≈ 0.0161. D = 1/62 ≈ 0.0161. Fused order: A, C, B, D. Note that C — third in dense, first in sparse — now sits beside A, and chunks found by only one retriever are kept as second-tier candidates rather than dropped.

Alternatives: min-max normalise each score list to [0, 1] and take a weighted sum (tunable, but brittle when one list has outliers); or let a Reranking cross-encoder re-score the union of both candidate lists, which is the most accurate and the most expensive. RRF is the default because it has one parameter and no assumptions about score distributions.

The shape is: run both retrievers with a generous k (say 50 each), fuse with RRF, optionally rerank the top 20–30, and pass the top 5–8 to the context builder. Both retrievers must see the same filters (Metadata Filtering) — a tenant filter applied only to the dense side leaks documents through the sparse side.

Tuning knobs: the per-retriever k, an optional weight favouring one list (multiply its contributions by 0.7 or 1.3), and BM25 tokenisation (lowercasing, keeping punctuation inside identifiers like E-4021, splitting camelCase for code). Query preprocessing helps too: extract any token that looks like an identifier and give it a mandatory-match boost on the sparse side.

Validate with a retrieval eval that includes both query styles. Hybrid should raise recall@10 relative to either retriever alone; if it lowers precision noticeably, the fused list is feeding junk to the LLM and you need a reranker or a stricter cutoff.