How do you implement deduplication in a NiFi flow?

Deduplication — preventing the same data from being processed more than once — is a common requirement. NiFi provides several mechanisms depending on scale, performance requirements, and what constitutes a duplicate.

DetectDuplicate processor: The simplest approach. It uses a Distributed Map Cache (a DistributedMapCacheClientService backed by a DistributedMapCacheServer) to store seen identifiers. For each incoming FlowFile, it evaluates a configurable Cache Entry Identifier (NiFi EL expression, e.g., ${filename} or ${sha256.hash}) and checks if that key already exists. Duplicates route to the duplicate relationship; new items go to non-duplicate. Cache entries can have a TTL (Age Off Duration) to forget old identifiers.

Content-based hashing: Use the HashContent processor to compute a SHA-256 hash of the FlowFile content and store it as an attribute, then use DetectDuplicate against the hash. This detects content-identical duplicates regardless of filename or metadata.

Database deduplication: For at-exactly-once semantics, track processed identifiers in a database table using PutDatabaseRecord with INSERT_IGNORE and a unique constraint on the identifier column. The database's ACID guarantees enforce uniqueness even under concurrent insertion.