Integration / Apache NiFi Interview Questions
How does NiFi clustering work and what is the role of ZooKeeper?
A NiFi cluster consists of multiple NiFi nodes that all run the same flow and collectively process data in parallel. Every node receives a copy of the flow definition and runs the same processors, but each node independently processes a subset of the FlowFiles — distributing the workload.
NiFi uses an embedded ZooKeeper (or an external ZooKeeper ensemble) for cluster coordination. ZooKeeper serves two critical roles:
Cluster Coordinator election: One node is elected Cluster Coordinator via ZooKeeper leader election. The Coordinator manages cluster state — which nodes are connected, which are heartbeating, and which are being disconnected. If the Coordinator fails, ZooKeeper elects a new one from the remaining nodes automatically.
Primary Node election: Separately, one node is designated as Primary Node. Processors configured with Execution: Primary Node Only run exclusively on the Primary Node. This is essential for processors that must not run concurrently — ListSFTP or QueryDatabaseTable would produce duplicate FlowFiles if all nodes ran them simultaneously.
All nodes communicate through the Site-to-Site protocol. The NiFi web UI can be accessed on any node and proxies canvas operations to all nodes transparently.
More Related questions...