Database / pgvector basics Interview Questions
How do you bulk-load vectors efficiently into pgvector?
Inserting vectors one row at a time with individual INSERT statements is the slowest possible approach. For large datasets (thousands to millions of rows), bulk loading strategies dramatically reduce ingestion time.
-- Method 1: Multi-row INSERT (batch inserts) INSERT INTO documents (content, embedding) VALUES ('Document 1', '[0.1,0.2,0.3]'), ('Document 2', '[0.4,0.5,0.6]'), ('Document 3', '[0.7,0.8,0.9]'); -- Send batches of 100-1000 rows per INSERT statement -- Method 2: COPY command (fastest for very large datasets) COPY documents (content, embedding) FROM '/path/to/data.csv' WITH (FORMAT TEXT, DELIMITER '\t'); -- CSV format: text<TAB>[v1,v2,v3,...] -- Method 3: psycopg2 executemany or copy_from (Python) # cur.executemany( # "INSERT INTO documents (content, embedding) VALUES (%s, %s)", # [(text1, emb1), (text2, emb2), ...] # ) -- Method 4: Disable index during load, rebuild after -- (Much faster for large bulk loads) -- Step 1: Create table WITHOUT index CREATE TABLE documents ( id BIGSERIAL PRIMARY KEY, embedding VECTOR(1536) ); -- Step 2: Bulk load all data COPY documents FROM '/data/embeddings.csv'; -- Step 3: Increase memory and workers for faster index build SET maintenance_work_mem = '4GB'; SET max_parallel_maintenance_workers = 8; -- Step 4: Build the index AFTER loading all data CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64);
More Related questions...