AI / LlamaIndex Interview Questions
Describe the ingestion pipeline in LlamaIndex?
The IngestionPipeline is LlamaIndex's way of chaining together the steps that turn raw Documents into indexed, embedded Nodes, so those steps don't need to be called manually one by one.
You configure it with a list of transformations, typically a node parser followed by an embedding model, and optionally metadata extractors in between. Running pipeline.run(documents=documents) applies every transformation in order and returns the resulting Nodes, which can then be inserted into an index or vector store.
from llama_index.core.ingestion import IngestionPipeline from llama_index.core.node_parser import SentenceSplitter pipeline = IngestionPipeline( transformations=[SentenceSplitter(chunk_size=512), embed_model] ) nodes = pipeline.run(documents=documents)
The pipeline can also attach a docstore for deduplication and a cache, so re-running it on unchanged documents skips redundant embedding calls.
An IngestionPipeline is configured primarily through:
Running pipeline.run(documents=documents) returns:
More Related questions...
