Vector Databases: What They Are and When You Actually Need One

The chatbot on this site does semantic search with no vector database at all — just a JSON file and a dot product. That's not a hack; it's the right call at its scale. But scale changes the math, and at some point a vector database stops being overkill and starts being necessary. The trick is knowing where that line is, which means understanding what a vector DB actually does.

The problem: searching by meaning

Traditional search matches keywords. "How do I speed up my API" won't match a document titled "Reducing endpoint latency" — no shared words, same meaning. Semantic search fixes this by representing text as embeddings: vectors where similar meanings land close together in space, regardless of shared words.

So "find relevant text" becomes "find the nearest vectors" — a geometry problem. Given a query vector, which stored vectors are closest?

Brute force is fine until it isn't

With a few hundred vectors, you just compare the query against every one and take the closest. That's exactly what my chatbot does:

index
  .map((chunk) => ({ chunk, score: cosine(query, chunk.vector) }))
  .sort((a, b) => b.score - a.score)
  .slice(0, k);

This is a linear scan — O(n) per query. At a few hundred or a few thousand vectors it's sub-millisecond and utterly fine. At ten million vectors, comparing against every one on every query falls apart. That's the wall a vector database is built to break through.

What a vector database actually adds

A vector DB (Pinecone, Weaviate, Milvus, or Postgres with pgvector) does three things a JSON file can't:

• Approximate Nearest Neighbor (ANN) indexing. Instead of checking every vector, it uses an index — typically HNSW, a navigable graph of vectors — to find the closest ones in roughly logarithmic time. It trades a tiny bit of accuracy (approximate, not exact) for an enormous speed win at scale.
• Metadata filtering. Combine similarity with structured filters: "closest chunks from this user's documents where lang = 'en'." Real RAG needs this constantly.
• Operational machinery. Persistence, updates, scaling, and concurrent access — the things a static file doesn't give you.

The honest threshold

Here's the decision, stripped of hype:

• Hundreds to a few thousand vectors, rebuilt occasionally → a file or an in-memory array. No database. Anything more is complexity you're carrying for no benefit.
• You already run Postgres and want vectors alongside relational data → pgvector. Add a vector column and an index to your existing database — no new service to operate. This is the sweet spot for most apps that outgrow a file.
• Millions of vectors, frequent updates, heavy filtered queries, high QPS → a dedicated vector database. This is where purpose-built ANN indexing and scaling genuinely pay off.

-- The middle-ground: vectors in the database you already have
CREATE EXTENSION vector;
ALTER TABLE chunks ADD COLUMN embedding vector(768);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
 
SELECT text FROM chunks ORDER BY embedding <=> $query LIMIT 5;

That <=> is pgvector's cosine-distance operator, and the HNSW index makes it fast — you get semantic search inside plain SQL, next to your relational data, with no extra infrastructure.

Don't cargo-cult the architecture

The mistake is copying the reference architecture — "RAG needs a vector database" — without checking whether your scale calls for it. A dedicated vector DB is a real service to provision, secure, monitor, and pay for. If your corpus fits in memory, you're adding a network hop and an ops burden to make a fast thing slower.

Start with the simplest thing that answers the query correctly: a file at small scale, pgvector when you outgrow it and already have Postgres, a dedicated vector database when the numbers genuinely demand it. The embedding-and-cosine idea is identical at every tier — only the indexing and infrastructure change as n grows.