Vector Storage

Nearest-neighbour search is trivial to state: given a query vector, return the k stored vectors with the highest cosine similarity. Brute force compares against every vector — O(n·d) per query. At 100k chunks × 768 dimensions that is ~77M multiply-adds, a few milliseconds on a modern CPU. At 50M chunks it is 38 billion, and brute force stops being an option.

A vector store is a system that keeps vectors plus metadata, offers approximate nearest-neighbour (ANN) search that is sub-linear in n, supports filtering by metadata, and handles inserts, updates, and deletes. The interesting engineering decisions are which of those you actually need.

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph where each vector links to its nearest neighbours; upper layers are sparse "express lanes", lower layers are dense. A query starts at the top, greedily hops toward the query vector, and descends layer by layer — like a skip list over a similarity graph. Search is roughly O(log n) hops, recall is typically 95–99%, and inserts are online. The cost is memory: the graph lives in RAM alongside the vectors, roughly 1.5–2× the raw vector size.

IVF (Inverted File index) clusters vectors with k-means into nlist cells (say 4096), stores each vector in its nearest cell, and at query time scans only the nprobe closest cells (say 32). It is cheaper in memory and builds fast, but recall drops when the query falls between cells, and the clustering must be retrained as the data distribution drifts. It is often combined with product quantisation (IVF-PQ) to compress vectors 8–32× at further recall cost.

Both trade recall for speed via tunable parameters (ef_search for HNSW, nprobe for IVF). Measure recall against brute force on a sample of real queries; a default that gives 90% recall silently loses one relevant chunk in ten before the LLM ever sees anything.

The pragmatic default for most teams is the database you already run. PostgreSQL with pgvector gives you a vector(1536) column, HNSW or IVFFlat indexes, cosine distance operators, and — critically — joins, transactions, row-level security, and the same backups and migrations as the rest of your data. A chunk row can carry tenant_id, acl, updated_at as normal columns and be filtered with normal WHERE clauses.

Dedicated vector stores (managed or self-hosted) win when scale or workload outgrows that: hundreds of millions of vectors, very high QPS, sharded indexes, built-in hybrid search with BM25, or multi-vector documents. They cost you an extra system to operate, a second source of truth to keep consistent, and usually weaker transactional guarantees.

Search engines (Elasticsearch / OpenSearch style) sit between: they already own your keyword index, add dense vectors with HNSW, and make Dense, Sparse & Hybrid Retrieval a single query. If you already run one, it is often the shortest path to hybrid search.

Real queries are never "nearest 5 vectors"; they are "nearest 5 vectors *that this user may see, from documents updated this year*". How a store combines the metadata filter with the ANN index matters enormously — a graph index that finds the 5 nearest and *then* filters may return zero results when the filter is selective. Metadata Filtering covers pre- vs post-filtering in depth; when choosing a store, check that filtered search keeps recall on your actual filters.

Consistency is the second overlooked property. When a document is updated, its old chunks must be deleted and new ones inserted; if the index and the source of truth are separate systems, they drift. A queue-based ingestion with doc_id + version on every chunk, idempotent upserts, and a periodic reconciliation job that compares counts per document is the usual shape. A vector column inside your transactional database avoids the drift entirely for small and medium corpora.

Run the numbers before adding infrastructure. Under ~100k chunks, a flat scan in pgvector or even in application memory returns in single-digit milliseconds. Under a few million, pgvector with HNSW on a reasonably sized instance serves hundreds of QPS. Most internal knowledge bases, support corpora, and per-tenant document sets never leave that range.

Escalate to a dedicated store when you can name the concrete limit you hit: index no longer fits in RAM, p95 latency exceeds budget under load, you need sharding across nodes, or you need a hybrid/multi-vector feature the database lacks. "Vector database" in the architecture diagram is not a requirement; sub-linear nearest-neighbour search with the right filters is.