Vector database

How it works

Each document passage is turned into an embedding: a list of hundreds or thousands of numbers that places it in a space where similar meanings sit close together. The vector database stores those lists alongside the original text and metadata such as source, date and access rights. A query is embedded the same way, and the database returns its nearest neighbors.

Comparing a query against millions of vectors one by one is slow, so these systems build approximate nearest neighbor indexes such as HNSW (opens in a new tab) (Malkov and Yashunin, 2016). They give up a sliver of accuracy to answer in milliseconds.

Options range from dedicated services (Pinecone, Qdrant, Weaviate, Milvus) to the pgvector (opens in a new tab) extension for Postgres, which many teams already run. It runs exact search by default and approximate search once you add an HNSW or IVFFlat index.

How businesses use it

A vector database is the retrieval layer under most RAG systems. It also powers semantic search on help centers, product recommendations and duplicate detection, for example spotting the same supplier entered 3 different ways.

For most companies with fewer than a few million passages, Postgres with pgvector is enough, and it keeps your data in a database you already back up and secure. A dedicated service is worth the cost at larger scale or with heavy filtering. On our RAG builds, the infrastructure sits on your account and is passed through at cost.

Where it goes wrong

Teams often pick the database first and the use case second. Which database you choose matters far less than how your documents are chunked and searched.

Watch for these problems instead:

• Stale data: a withdrawn policy stays searchable because nobody wired deletions into the index
• Missing filters: without metadata for department or permissions, results leak across teams
• Model changes: switching embedding models means re-embedding every document, because vectors from different models cannot be compared
• Exact terms: vector similarity is weak on SKUs and error codes, so pair it with hybrid search

A vector database is a search component. It does not decide what is true, and it returns its closest match even when nothing relevant exists.