How does NiFi achieve guaranteed delivery and what are its durability guarantees?

NiFi's architecture is specifically designed to provide guaranteed delivery — once data enters NiFi, it will not be silently lost due to hardware failure, software crash, or network issues. Several design decisions work together to achieve this.

Persistent connection queues: FlowFiles in connection queues are tracked in the FlowFile Repository's Write-Ahead Log. On restart after a crash, NiFi replays the WAL to restore every FlowFile to its exact queue position before the crash.

Immutable content repository: FlowFile content is written to disk before the FlowFile is considered active. Content is never deleted until all references are removed. Even if NiFi crashes mid-write, the old content version is preserved.

Transactional processor sessions: Each processor invocation runs within a ProcessSession. All changes within a session are committed atomically. If the processor throws an exception before committing, the session is rolled back: all FlowFiles return to their input queues as if nothing happened.

Site-to-Site acknowledgment: Data transferred via S2S is acknowledged by the receiver before the sender removes FlowFiles from its queues. If the receiver crashes before acknowledgment, the sender retains the data and retries.

The practical implication: NiFi provides at-least-once delivery by default. Under failure and retry scenarios, a FlowFile may be processed more than once. Achieving exactly-once requires idempotent downstream systems or explicit deduplication logic in the flow.