Vector Search: Embeddings, Similarity and ANN

An embedding model maps a piece of text (or an image, or code) to a fixed-length vector — 384 to 3,072 floats — such that texts with similar meaning land near each other. The model is the whole semantics; two vectors from different models are not comparable. A document is split into chunks because one vector for forty pages means nothing; each chunk gets its own vector, and the chunk is the unit of retrieval. Chunk size is the most consequential parameter in the pipeline — see the agentic side in Ingestion: Parsing & Chunking.

Cosine similarity is the cosine of the angle between two vectors: direction only, magnitude ignored. The default for text. Dot product is direction and magnitude; on normalised vectors it equals cosine and is cheaper. Euclidean (L2) distance is the straight-line distance; natural for image features, unusual for text. Pick the one the embedding model was trained for — the model card says — and use it consistently.

Exact search scores every vector and sorts: O(n × d). Fine to a few hundred thousand vectors, hopeless at fifty million. That is where approximate nearest neighbour comes in.

HNSW (Hierarchical Navigable Small World) builds a layered graph: a sparse top layer for coarse navigation, denser layers below. Search enters at the top, greedily walks to the nearest node, drops a layer, repeats. O(log n) hops, each visiting a few neighbours. Two knobs: M (edges per node — memory and recall) and ef_search (candidates kept during search — latency and recall). Build is slow and memory-hungry; the index lives in RAM. IVF (inverted file) clusters the vectors, and a query searches only the nearest few clusters; cheaper to build, lower recall, sensitive to data drift.

Both are approximate: they may miss the true nearest neighbour. Recall of 95–99% is typical and tunable. Whether that is acceptable depends on the use — for RAG it almost always is, because the top-5 need only contain *a* good chunk, not *the* best one.

Real queries carry constraints: this tenant, this language, documents updated after a date. Pre-filtering applies the constraint before the ANN search, which can starve the graph walk of reachable candidates and collapse recall. Post-filtering takes the top-k and then filters, which can leave you with zero results. Engines handle this differently — pgvector’s HNSW with a WHERE clause, dedicated stores with filtered graph traversal — and it is the first thing to test with your actual filter selectivity.

Embeddings are bad at exact tokens: an error code, a product SKU, a name. Hybrid search runs a keyword (BM25) query alongside the vector query and fuses the two ranked lists, usually with reciprocal rank fusion. Most production retrieval is hybrid — see Dense, Sparse & Hybrid Retrieval.

PostgreSQL + pgvector: the vectors sit next to the rows they describe; filters are SQL; transactions cover both; one system to run. Good to tens of millions of vectors. Dedicated vector database: built for hundreds of millions to billions, sharded ANN, tuned filtered search — and a second system with its own consistency lag from your source of truth. Search engine: the strongest hybrid story if you already run one. The rule is the same as everywhere: start in Postgres, move when a measured number says so. See Vector Storage for the agentic-side view.