Integration / Apache NiFi Interview Questions
What is the difference between GetFile and ListFile + FetchFile processors?
Both approaches ingest files from a local filesystem, but they differ in architecture, parallelism, and operational characteristics.
GetFile: The older, simpler, single-processor approach. It lists a directory, picks up files matching the configured filter, moves or deletes the source file atomically, and produces a FlowFile with the file content. Critical limitation: it is not safe to run with multiple concurrent tasks because two threads could attempt to process the same file simultaneously — its filesystem rename locking is not atomic on all filesystems or NFS mounts. On a cluster, GetFile should run on the Primary Node only.
ListFile + FetchFile: The modern, recommended approach. ListFile scans the directory and emits one FlowFile per found file containing only metadata attributes (filename, path, size, last modified). It uses State Management to track already-listed files. FetchFile then reads the actual file content from disk. This separation enables:
- Parallel fetching: multiple concurrent FetchFile tasks read files simultaneously
- Clear separation of concerns: listing happens once; fetching can be retried independently per file
- Works correctly in clustered NiFi without Primary Node restriction on the fetch step
GetFile remains appropriate for simple single-node use cases. For new development and cluster deployments, ListFile + FetchFile is preferred.
More Related questions...