What are vector databases and how do they enable semantic search in RAG pipelines?

Vector databases store numerical vector representations (embeddings) of documents and enable fast approximate nearest-neighbour (ANN) search — retrieving the vectors most similar to a query vector, typically measured by cosine similarity or inner product. This is the retrieval backbone of every RAG system.

The workflow has two phases. Indexing: each document chunk is passed through an embedding model (e.g. text-embedding-3-small or BAAI/bge-small-en-v1.5) to produce a fixed-size vector; the vector plus metadata is stored in the vector DB. Querying: the user's query is embedded with the same model, and the DB returns the k chunks whose vectors are closest to the query vector. Popular options include FAISS (in-memory, open-source), Chroma (embedded, easy local dev), and Pinecone / Weaviate (managed cloud).

# ── FAISS: local in-memory vector search
import faiss
import numpy as np
from openai import OpenAI

client = OpenAI()

def embed(texts: list[str]) -> np.ndarray:
    resp = client.embeddings.create(
        model='text-embedding-3-small',
        input=texts
    )
    return np.array([d.embedding for d in resp.data], dtype='float32')

docs = [
    'Python was created by Guido van Rossum in 1991.',
    'The Eiffel Tower is located in Paris, France.',
    'Machine learning is a subset of artificial intelligence.',
]

doc_vecs = embed(docs)          # (3, 1536)
faiss.normalize_L2(doc_vecs)    # normalise for cosine similarity via dot product

index = faiss.IndexFlatIP(doc_vecs.shape[1])  # inner product index
index.add(doc_vecs)

query_vec = embed(['Who invented Python?'])
faiss.normalize_L2(query_vec)
distances, indices = index.search(query_vec, k=2)  # top-2 results
for i in indices[0]:
    print(docs[i])
# Python was created by Guido van Rossum in 1991.  <- top match

# ── Chroma: persistent local vector DB
import chromadb

chroma = chromadb.PersistentClient(path='./chroma_db')
collection = chroma.get_or_create_collection('my_docs')
collection.add(
    documents=docs,
    ids=[f'doc_{i}' for i in range(len(docs))],
)
results = collection.query(query_texts=['Who invented Python?'], n_results=2)
print(results['documents'])