Explain the internal working of the IngestionPipeline caching mechanism?

The IngestionPipeline's cache exists to avoid redoing expensive work, mainly embedding calls, when the pipeline is re-run on data it has already processed.

Internally, the IngestionCache computes a hash from each input Node's content combined with the specific transformation and its configuration being applied, for example a particular SentenceSplitter(chunk_size=512) instance. That hash acts as a cache key. When the pipeline runs, it checks whether that exact combination of content and transformation has produced output before; if so, it reuses the cached result instead of recomputing it, and if not, it runs the transformation and stores the new output under that hash.

This is combined with a docstore that tracks document identity and content hashes for deduplication at the document level, so unchanged documents can be skipped entirely on re-ingestion, while the transformation-level cache handles the finer-grained case where a document changed but a specific downstream step's output for a given Node hasn't. Together they mean re-running a pipeline on a mostly-unchanged corpus is cheap, since only the genuinely new or modified content triggers fresh embedding calls.