Vector databases 101
What a vector database does, how nearest-neighbour search works, and how to pick one.
If you are building a Retrieval-Augmented Generation (RAG) system, a way of giving a language model relevant facts to work from, you will meet the term “vector database” quickly. It sounds intimidating. It is not. This guide explains what one does, how it finds things, and how to pick one, without assuming you have seen any of this before.
What a vector database actually stores
Start with the problem. You have a pile of text: support articles, product manuals, research papers, whatever your system needs to know. You want to find the passages that answer a given question. A normal keyword search looks for matching words. That misses a lot, because people ask questions using different words than the answer uses. Someone types “my card was declined” and the helpful article is titled “payment failures”.
A vector database solves this by working with meaning rather than exact words. First you convert each passage into an embedding, a list of numbers (often a few hundred or a few thousand of them) that captures what the text means. A model called an embedding model produces these numbers. The useful property is that passages with similar meanings get similar numbers. “My card was declined” and “payment failures” end up close together, even though they share no words.
You can picture each embedding as a point in space, not the flat space of a map but a space with hundreds of dimensions. The exact geometry does not matter for using one. What matters is the idea: similar meanings sit near each other, different meanings sit far apart.
A vector database stores all these points and, crucially, is built to answer one question fast: “given this new point, which stored points are nearest?”
How nearest-neighbour search works
When a question arrives, your system turns it into an embedding using the same model, then asks the database for the closest stored passages. This is called nearest-neighbour search: find the points nearest to the query point. Those nearby passages are your best candidates for an answer, and you hand them to the language model as context.
The catch is scale. If you have 10 million passages, checking the query against every single one is slow and expensive. Comparing against all of them, called exact nearest-neighbour search, does not hold up once your collection grows.
So vector databases use approximate nearest-neighbour search, usually shortened to ANN. Instead of checking everything, they organise the points in advance into a structure (a graph, a set of clusters, or a tree) that lets the search skip most of the collection and still land on the closest points nearly every time. “Approximate” means it might occasionally miss the true closest match by a hair, but it returns excellent results in a fraction of the time. For RAG, that trade is almost always worth it: you get answers in milliseconds instead of seconds, and the tiny loss in precision rarely changes which passages you retrieve.
You will often see a setting that lets you tune this balance. A sensible default is fine when you are starting out.
Metadata filtering
Meaning is not the only thing you care about. Often you want to narrow results by plain facts: only documents in English, only articles published after 2024, only pages for the product the customer actually owns. These facts are called metadata, the labels you attach to each passage alongside its embedding.
Good vector databases let you combine the two: “find the passages closest in meaning to this question, but only among those tagged language: en and product: pro.” This is metadata filtering, and it matters more than people expect. Without it, a question about your Pro plan might pull in a passage about the Free plan simply because the two read similarly. With it, you keep answers accurate and scoped. When you compare options, check how well each one filters, because some handle it far more efficiently than others.
How to choose one
There is no single best vector database, only the right fit for your situation. A few honest questions to work through:
Open source or managed?
An open-source vector database (examples include Qdrant, Weaviate, Milvus, and the pgvector extension for PostgreSQL) runs on your own machines. You control the data and pay only for the hardware, but you also run, update, and back it up yourself. A managed service runs it for you for a monthly fee: less to operate, less control, an ongoing cost. If you already run a PostgreSQL database, adding pgvector to it is often the simplest first step, because it avoids introducing a whole new system.
How much data, and how fast?
A few thousand passages behave very differently from a few hundred million. At small scale, almost anything works well, and you should optimise for ease rather than raw speed. At large scale, indexing strategy, memory use, and cost dominate the decision, so test with data shaped like yours before committing.
What else do you need?
Consider metadata filtering (covered above), whether you can update and delete passages easily, and how it fits the tools you already use. Do not choose on benchmark numbers alone. The fastest option on paper can still be the wrong one if it is painful to run day to day.
A note of caution: every vendor will tell you theirs is fastest and simplest. Treat those claims as a starting point for your own testing. Your data and your questions are what count.
Next steps
You now know the shape of it: a vector database stores meaning as points, finds the nearest ones quickly with approximate nearest-neighbour search, and lets you filter by plain facts. The best next move is to try it with real content. Browse the RAG Repo directory to find open datasets you can practise on, and keep the glossary open for any term that trips you up. Get a simple search working end to end, then grow from there.