Database / LanceDB Interview questions
1. What is LanceDB?
LanceDB is an open-source, embedded vector database built for AI applications, providing vector similarity search, full-text search, and SQL-style filtering over the same table without needing a separate server process to deploy or manage. It runs directly inside your application process as a lib...
2. What is the purpose of LanceDB?
LanceDB exists to give AI applications a single place to store and query vectors, raw multimodal data (text, images, audio), and structured metadata together, instead of stitching together a separate vector index, a separate blob store, and a separate metadata database. Traditional setups for AI ...
3. What is the Lance columnar format?
Lance is the open-source columnar file and table format that LanceDB is built on — conceptually similar in role to Parquet, but specifically optimized for the access patterns AI and machine learning workloads need, particularly fast random access to individual rows or vectors rather than on...
4. What are the key features of LanceDB?
LanceDB combines several capabilities that are often split across separate tools in a typical AI retrieval stack, all built around one shared columnar table. Feature What it Provides Embedded architecture Runs in-process, no separate server to deploy Multimodal storage Vectors, text, images, and ...
5. What is an embedded vector database?
An embedded database is one that runs as a library directly inside your application's own process, rather than as a separate server your application connects to over a network — the same architectural pattern SQLite uses for relational data, applied here to vector search. Because there's no...
6. What is Apache Arrow, and how does LanceDB use it?
Apache Arrow is an open standard for representing tabular, columnar data in memory in a way that many different tools and languages can share directly, without needing to serialize and deserialize data when passing it between them. LanceDB uses Arrow as its in-memory data representation, which me...
7. Define a table in LanceDB?
A table in LanceDB is the core unit of data storage — a columnar collection of rows sharing a defined schema, conceptually similar to a table in a relational database, except its columns commonly include a fixed-size vector column alongside ordinary scalar columns like strings or numbers. i...
8. What is a vector embedding?
A vector embedding is a list of floating-point numbers that represents a piece of data — text, an image, audio — in a way that captures its meaning or content, positioned in a high-dimensional space such that similar items end up close together and dissimilar items end up far apart. E...
9. What are the supported languages/SDKs for LanceDB?
LanceDB provides official SDKs built on top of its shared Rust core, so the same underlying engine and file format is accessible from multiple languages without each language reimplementing the storage and query logic separately. Language / SDK Notes Python The most mature and widely used SDK Nod...
10. How do you create a table in LanceDB?
Creating a table in LanceDB typically means connecting to a database directory (local or object storage), then calling a create method with either an initial batch of data or an explicit schema. import lancedb from lancedb.pydantic import LanceModel, Vector db = lancedb . connect( "./my_lancedb" ...
11. What is ANN (Approximate Nearest Neighbor) search?
Approximate Nearest Neighbor search is a technique for finding vectors close to a query vector without exhaustively comparing the query against every single vector in the dataset, trading a small amount of recall accuracy for a large gain in search speed. An exact nearest-neighbor search (sometim...
12. What is the IVF_PQ index in LanceDB?
IVF-PQ (Inverted File index with Product Quantization) is one of LanceDB's supported ANN index types for vector columns, combining two complementary techniques to make approximate similarity search fast even over very large datasets stored on disk. The IVF part partitions the vector space into a ...
13. What is full-text search in LanceDB?
Full-text search (FTS) in LanceDB lets you search a text column for keyword matches using a BM25-based ranking algorithm, complementing vector similarity search with traditional, exact keyword-based retrieval on the same table. table.create_fts_index("text") results = table.search("machine learni...
14. What is hybrid search in LanceDB?
Hybrid search combines vector similarity search and full-text (keyword) search in a single query, then merges and re-ranks the two result sets into one final ranked list, aiming to get the best of both: semantic understanding from vector search and precise keyword matching from full-text search. ...
15. Define the embedding function registry in LanceDB?
The embedding function registry is LanceDB's mechanism for attaching an embedding model directly to a table's schema, so that inserting raw data (like text) automatically generates and stores the corresponding vector, without the application needing to call an embedding model explicitly before ev...
16. What is schema evolution in LanceDB?
Schema evolution refers to LanceDB's ability to add, rename, retype, or drop columns on an existing table without needing to rewrite the entire dataset from scratch, which is possible specifically because of the Lance format's columnar storage design. table.add_columns({"category": "cast(NULL as ...
17. List the storage backends supported by LanceDB?
LanceDB can store its underlying Lance-format data files on a local filesystem or directly on cloud object storage, without needing a separate data-loading step to move data between the two. Backend Typical Use Local filesystem Development, prototyping, single-machine production workloads Amazon ...
18. What is a scalar index in LanceDB?
A scalar index is an index built on a regular, non-vector column — like a string, number, or boolean field — to speed up filtering queries (a WHERE clause) against that column, distinct from the vector (ANN) indices used for similarity search. table.create_scalar_index("category") res...
19. What is versioning in LanceDB?
Versioning means every write to a LanceDB table — adding rows, updating them, deleting them, or changing the schema — creates a new, immutable version of the table rather than overwriting existing data in place, giving every table a complete, git-like history of changes. print(table ....
20. How do you connect to a LanceDB database?
Connecting to a LanceDB database means calling lancedb.connect() (or its async equivalent) with a URI pointing at where the database's tables should live — a local directory, a cloud object storage path, or a LanceDB Cloud connection string. import lancedb # Local filesystem db = lancedb . ...
21. What is the difference between LanceDB and Pinecone?
Both are used for vector similarity search, but they differ fundamentally in architecture: LanceDB is an embedded, open-source library you run yourself, while Pinecone is a fully managed, closed-source cloud service you connect to over a network. LanceDB Pinecone Open-source, embeddable library; ...
22. What is the difference between LanceDB and Chroma?
LanceDB and Chroma are both open-source, embedded-first vector databases with a similar "get started instantly, no server required" philosophy, but they differ in underlying storage design and how they scale toward production. LanceDB Chroma Built on the Lance columnar lakehouse format. Uses its ...
23. What is the difference between LanceDB OSS and LanceDB Cloud?
LanceDB OSS (open-source software) is the free, self-hosted embedded library anyone can run locally or point at their own object storage, while LanceDB Cloud is a managed offering that handles infrastructure, scaling, and concurrent multi-process access on the customer's behalf. LanceDB OSS Lance...
24. Why is LanceDB well suited for multimodal AI data?
LanceDB is designed so that text, vector embeddings, images, audio, and other data types can all live as columns within the same table, rather than requiring separate storage systems per data type that then have to be joined or synchronized at query time. Because the underlying Lance format is a ...
25. How does the Lance format differ from Parquet?
Both are open, columnar file formats, but they were optimized for different access patterns: Parquet was designed primarily for efficient large, sequential analytical scans, while Lance was designed to also support fast random access to individual rows, which matters much more for vector search a...
26. Explain how IVF_PQ indexing works internally?
IVF-PQ is built and used in two distinct phases — an index-build phase that happens once (or periodically, as data grows) and a query phase that runs for every search — and understanding both is what makes the speed/accuracy trade-off concrete rather than abstract. flowchart TD A[Vect...
27. What is the difference between IVF_PQ and HNSW indexing in LanceDB?
Both are ANN index types LanceDB supports for vector columns, but they take fundamentally different approaches to narrowing a search space, which leads to different trade-offs in memory usage, build time, and query speed. IVF-PQ HNSW Cluster-based partitioning plus vector compression. Graph-based...
28. How does LanceDB implement hybrid search using reranking?
Hybrid search in LanceDB runs a vector query and a full-text query independently against the same table, then hands both ranked result lists to a reranker component whose job is to merge them into one final, unified ranking rather than simply picking one list over the other. flowchart TD A[Hybrid...
29. What is Reciprocal Rank Fusion (RRF), and how is it used in LanceDB?
Reciprocal Rank Fusion is a rank-based method for combining multiple ranked result lists into one unified ranking, by giving each item a score based on the reciprocal of its position (rank) in each list it appears in, then summing those reciprocal scores across all the lists it's present in. from...
30. Explain the lifecycle of a write operation in LanceDB (versioning)?
Every write to a LanceDB table — whether it's adding rows, updating them, deleting them, or altering the schema — goes through the same fundamental lifecycle: new data files are written, and a new manifest describing the table's state is committed atomically, advancing the table to a ...
31. How do you perform time travel queries in LanceDB?
Time travel means querying a LanceDB table as it existed at a specific past version, rather than its current latest state, by putting the table into a pinned, read-only mode pointed at that version. print(table . version) # e.g. 5, the latest version table . checkout( 2 ) # pin the table to versi...
32. What is the difference between checkout and restore in LanceDB?
Both operations move a table's view to a past version, but they differ in whether that move is temporary and read-only or permanent and writable going forward. checkout() restore() Pins the table to a past version for reading. Creates a new version whose data matches a past version. Read-only whi...
33. What are tags in LanceDB versioning?
Tags are named, human-readable references to a specific table version — conceptually similar to Git tags — letting you refer to a version by a meaningful name like "pre-migration" or "v2-eval-baseline" instead of remembering an opaque numeric version. table.tags.create("stable-release...
34. What are branches in LanceDB, and how do they differ from tags?
Branches are isolated, writable lines of history forked from a table's main history (or from a specific past version), letting you make and accumulate changes separately from the production line of history without affecting reads against the main branch. Tags Branches Read-only reference to one e...
35. How does LanceDB handle deletes internally?
A delete in LanceDB is a non-destructive operation at the storage level: rather than immediately erasing the deleted rows' data, it writes a new version with deletion markers flagging those rows as removed, while the actual underlying data remains present on disk until a later, explicit cleanup s...
36. Explain the internal working of LanceDB's zero-copy data access via Arrow?
Zero-copy access means that when data moves between LanceDB and another Arrow-compatible tool, the underlying bytes in memory don't need to be copied or re-serialized into a different format — both sides simply agree on and read the same memory layout directly. flowchart TD A[Lance file on ...
37. How do you integrate LanceDB with LangChain for RAG?
LangChain provides a LanceDB vector store class that wraps a LanceDB table behind LangChain's standard vector store interface, letting LanceDB slot into an existing LangChain RAG pipeline as the retrieval backend with minimal glue code. from langchain_community.vectorstores import LanceDB import ...
38. What is the role of DataFusion in LanceDB's query execution?
Apache DataFusion is a Rust-based, extensible query execution engine that LanceDB uses under the hood to plan and execute queries — particularly SQL-style filtering and more complex query expressions — rather than LanceDB implementing its own query planner and execution logic from scr...
39. How do you implement custom embedding functions in LanceDB?
When a built-in embedding provider in the registry doesn't cover a specific model, LanceDB lets you implement the EmbeddingFunction interface directly and register it, so the custom function gets the same automatic embed-on-insert and embed-on-query behavior as any built-in provider. from lancedb...
40. When should you use a scalar index versus a vector index?
The two index types accelerate fundamentally different kinds of query, and picking the right one for a given column comes down to what kind of question you're asking against it — similarity, or exact/range matching. Use a Scalar Index When Use a Vector Index When Filtering on exact values, ...
41. How do you optimize a LanceDB table for query performance through compaction?
Compaction consolidates many small data files (accumulated from repeated small writes over a table's version history) into fewer, larger files, which improves both scan performance and vector index quality, since a fragmented table with many tiny files has more overhead per query than a well-cons...
42. What is prefiltering vs postfiltering in LanceDB queries?
When a query combines vector search with a metadata filter, there are two possible orders to apply them in, and LanceDB supports both, with different performance and correctness trade-offs depending on how selective the filter is. Prefiltering (default) Postfiltering Filter applied to narrow the ...
43. Explain the internal working of product quantization (PQ) in vector indexing?
Product Quantization compresses a high-dimensional vector by splitting it into several smaller sub-vectors and separately approximating each sub-vector using a small, shared codebook, dramatically reducing the storage and computation cost of comparing vectors at the price of some precision loss. ...
44. How does LanceDB support multi-process concurrent access?
Because LanceDB is fundamentally an embedded library reading and writing files directly, multi-process concurrent access depends on the underlying storage backend's own concurrency guarantees, with the specifics differing between local filesystem use and object storage. On local filesystem storag...
45. What are the trade-offs of running LanceDB embedded versus as a managed cloud service?
Choosing between self-hosted LanceDB OSS and the managed LanceDB Cloud/Enterprise offering trades operational control for operational convenience, and the right choice depends heavily on scale, team capacity, and data governance requirements. Operational burden: OSS means you manage scaling, back...
46. Explain the internal working of the manifest and commit protocol in Lance?
Every version of a Lance table is described by a manifest — a metadata file listing exactly which data files make up that version's state, along with schema information — and advancing to a new version means atomically committing a new manifest that supersedes the previous one. sequen...
47. How do you implement multimodal search across text and images in LanceDB?
Multimodal search across text and images relies on using a shared embedding model — like CLIP — that maps both modalities into the same vector space, so a text query's embedding can be compared directly against stored image embeddings (or vice versa) using ordinary vector similarity s...
48. What is the role of object storage (S3/GCS) in LanceDB's architecture?
Object storage serves as the durable, scalable source of truth for a LanceDB table's data and version history, letting compute (the application process running queries) remain stateless and decoupled from where the actual data physically lives — a foundational pattern often called separatin...
49. How do you monitor and troubleshoot slow vector search queries in LanceDB?
Diagnosing slow vector queries generally starts with confirming the basics — is an appropriate index actually present and being used — before moving on to more nuanced tuning of index parameters and query structure. Check whether an ANN index exists on the vector column: without one, ...
50. Explain the execution flow of a RAG pipeline built with LanceDB as the retrieval layer?
A typical retrieval-augmented generation pipeline using LanceDB moves through an ingestion phase (done once, or incrementally as new content arrives) and a query-time phase (done for every user request), with LanceDB serving as the shared retrieval layer connecting the two. flowchart TD subgraph ...
