Integration / Apache NiFi Interview Questions
1. What is Apache NiFi and what problem does it solve?
Apache NiFi is an open-source data integration and dataflow automation platform originally developed by the NSA under the codename Niagarafiles and donated to the Apache Software Foundation in 2014. It provides a web-based graphical interface for designing, controlling, and monitoring data flows ...
2. What is a FlowFile in Apache NiFi?
A FlowFile is the fundamental unit of data in Apache NiFi. Every piece of data moving through a flow is represented as a FlowFile with two distinct parts. Attributes : A map of key-value string pairs acting as metadata. Every FlowFile has core attributes automatically assigned — uuid (globally un...
3. What are the three NiFi repositories and what does each store?
Apache NiFi uses three on-disk repositories, each serving a distinct durability and query purpose. FlowFile Repository : Stores the current state of all active FlowFiles — their attributes and a pointer (content claim) to where content lives in the content repository. Uses a Write-Ahead Log (WAL)...
4. What is a Processor in Apache NiFi and what are the main processor categories?
A Processor is the fundamental building block of a NiFi data flow. Each processor performs one specific operation on FlowFiles: fetching from a source, transforming content, routing on attributes, writing to a destination, and so on. Processors are connected via Connections to form a directed dat...
5. What is a Connection in NiFi and how does back-pressure work?
A Connection in NiFi is a directed, persistent queue that links the output of one processor to the input of another. FlowFiles physically queue here when the downstream processor cannot keep up. Each connection carries one or more relationships from the upstream processor (e.g., success , failure...
6. What is NiFi Expression Language and where can it be used?
NiFi Expression Language (EL) is a built-in expression engine enabling dynamic evaluation of FlowFile attribute values within processor property configurations. Instead of hardcoded values, you write expressions evaluated at runtime against each FlowFile's attributes using the syntax ${attribute....
7. What is data provenance in Apache NiFi and how do you access it?
Data provenance in NiFi is the complete, immutable audit trail of everything that happens to every FlowFile from when it enters until it leaves or is dropped. NiFi records a provenance event automatically for every significant action — no explicit configuration is required. Provenance event types...
8. What is a Process Group in NiFi and why is it used?
A Process Group is a named container that groups processors, connections, funnels, and other components into a single unit on the NiFi canvas. It appears as a rectangle; double-clicking opens it to reveal its internals. Process Groups can be nested, enabling hierarchical flow organization. Organi...
9. What is NiFi Registry and how does it integrate with NiFi?
NiFi Registry is a complementary subproject that provides centralized storage, management, and versioning of NiFi flow definitions. It functions as a version control system for Process Group configurations — similar in concept to Git for source code. NiFi Registry organizes flows into Buckets — n...
10. How does NiFi clustering work and what is the role of ZooKeeper?
A NiFi cluster consists of multiple NiFi nodes that all run the same flow and collectively process data in parallel. Every node receives a copy of the flow definition and runs the same processors, but each node independently processes a subset of the FlowFiles — distributing the workload. NiFi us...
11. What is a Controller Service in NiFi and how is it different from a Processor?
A Controller Service is a shared, reusable service that processors within a Process Group (or across the entire NiFi instance) can reference in their configurations. Where a Processor performs work on individual FlowFiles, a Controller Service provides a shared capability — a database connection ...
12. What is the GenerateTableFetch and QueryDatabaseTable pattern for incremental database ingestion?
QueryDatabaseTable and GenerateTableFetch are the two primary patterns for incrementally ingesting data from relational database tables. Each has different performance characteristics and use cases. QueryDatabaseTable : A simpler, single-processor approach. It issues a SELECT query using a config...
13. What is the Record-based processing model in NiFi and why is it preferred?
NiFi's record-based processing model treats FlowFile content as a structured stream of records rather than an opaque blob. A record is one logical row — one JSON object, one CSV line, one Avro record, one database row. Record-aware processors operate on individual records within a FlowFile, enabl...
14. What is State Management in NiFi and what types of state scope exist?
State Management is NiFi's built-in mechanism for processors and controller services to persistently store small amounts of key-value data that survive processor restarts and NiFi restarts. Without state management, a processor like QueryDatabaseTable would forget the last ingested timestamp ever...
15. What is NiFi Site-to-Site (S2S) and when do you use it?
NiFi Site-to-Site (S2S) is a native protocol for transferring FlowFiles directly between NiFi instances — between two standalone NiFi servers, between nodes in a cluster, or between NiFi and MiNiFi agents. It operates over HTTP/HTTPS or raw TCP sockets and provides end-to-end guaranteed delivery,...
16. What is NiFi and how does it relate to Apache NiFi?
MiNiFi (Minimum NiFi) is a lightweight subproject of Apache NiFi designed specifically for edge data collection on resource-constrained devices — IoT sensors, industrial controllers, embedded systems, and edge servers. It implements a subset of NiFi's capabilities with a dramatically reduced foot...
17. What is NiFi Parameter Context and how does it differ from Variables?
A Parameter Context is a named collection of key-value parameters applied to a Process Group to externalize configuration values from the flow definition. Instead of hardcoding a Kafka broker address or database URL inside processor properties, you reference a parameter with the syntax #{paramete...
18. How does NiFi handle security — TLS, authentication, and authorization?
NiFi provides a comprehensive security model covering transport encryption, user authentication, and fine-grained authorization. TLS / Transport Encryption : NiFi can be configured to serve its UI and API exclusively over HTTPS. TLS is configured in nifi.properties using a keystore (server certif...
19. What is the NiFi NAR (NiFi Archive) classloading model?
The NAR (NiFi ARchive) is the extension packaging format for NiFi components: processors, controller services, and reporting tasks. A NAR file is similar to a JAR but includes a special manifest that declares its dependencies and classloader parent chain. The NAR classloading model solves the dep...
20. What are Reporting Tasks in NiFi and what are common use cases?
Reporting Tasks are NiFi extension components that run on a scheduled basis to collect and report metrics, bulletin events, and operational data from the NiFi instance itself — not from FlowFiles. They operate at the NiFi system level rather than the data flow level, making them the primary tool ...
21. How do you handle errors and failures in a NiFi flow?
NiFi provides several mechanisms for handling failures gracefully, ensuring that failed FlowFiles are not silently lost and that problems are visible to operators. Failure Relationships : Most processors emit FlowFiles that cannot be processed to a failure relationship. Always connect this relati...
22. What is the SplitText processor and how do you control split behavior?
SplitText is a NiFi processor that splits a FlowFile containing multiple lines of text into multiple smaller FlowFiles, each containing a configurable number of lines. It is the workhorse for splitting large text, CSV, or log files into processable chunks before parallel processing. Key configura...
23. What is the MergeContent processor and how is it used?
MergeContent is a NiFi processor that combines multiple FlowFiles into a single FlowFile. It is the counterpart to processors like SplitText and SplitJSON, enabling a scatter-gather pattern: split a large FlowFile into pieces for parallel processing, then merge the results back together. MergeCon...
24. What is the InvokeHTTP processor and what are key configuration considerations?
InvokeHTTP is NiFi's most flexible HTTP client processor. It sends HTTP requests to configurable URLs using any HTTP method (GET, POST, PUT, PATCH, DELETE) and routes the response to different relationships based on the HTTP response code. It is the Swiss-army knife for REST API integration in Ni...
25. What is the PublishKafka and ConsumeKafka processor pair and what are key configuration options?
PublishKafka and ConsumeKafka (and their record-aware variants PublishKafkaRecord and ConsumeKafkaRecord) are NiFi's integration points with Apache Kafka. ConsumeKafka : Subscribes to one or more Kafka topics using the Kafka consumer group protocol. Key properties include: Kafka Brokers (bootstra...
26. What is the ExecuteScript processor and what scripting languages does it support?
ExecuteScript is NiFi's escape hatch for custom logic that cannot be expressed with built-in processors. It allows you to write arbitrary script code that executes within the NiFi processor lifecycle — accessing incoming FlowFiles, creating new FlowFiles, modifying attributes, reading and writing...
27. What is the JoltTransformJSON processor and how do you use it?
JoltTransformJSON is a NiFi processor that transforms JSON content using the JOLT (JSON to JSON transformation) library. JOLT uses declarative JSON specifications (JOLT specs) to describe how an input JSON document should be restructured — renaming fields, changing nesting structure, filtering ar...
28. What is the PutDatabaseRecord processor and how does it differ from ExecuteSQL?
PutDatabaseRecord writes structured records from a FlowFile into a relational database table using JDBC. It is the write counterpart to QueryDatabaseTable and ExecuteSQL. Unlike ExecuteSQL — which executes arbitrary SQL statements — PutDatabaseRecord works with the NiFi record model: it reads rec...
29. What is the ListSFTP and FetchSFTP processor pattern and how does it work?
ListSFTP and FetchSFTP implement a two-stage pattern for ingesting files from SFTP servers, separating listing from fetching. This design also appears for S3 (ListS3/FetchS3Object), Azure Blob Storage, HDFS, and local filesystems. ListSFTP : Connects to the SFTP server and lists files in the conf...
30. What is the LookupRecord processor used for?
LookupRecord is a record-aware NiFi processor that enriches records within a FlowFile by looking up values from an external source — a database, a distributed map cache, a REST API, or a file-based lookup table — and adding the result as a new field in each record. LookupRecord works with three c...
31. What is the PartitionRecord processor and what is a common use case?
PartitionRecord is a record-aware NiFi processor that reads records from an input FlowFile and groups them into separate output FlowFiles based on one or more field values. All records sharing the same value for the partition field(s) go to the same output FlowFile; records with different values ...
32. What is the ConvertRecord processor and how is it used for format conversion?
ConvertRecord is a NiFi processor that converts FlowFile content from one data format to another using the NiFi record model. Its sole job is to read records using one format and write them out in another. The conversion logic lives entirely in the RecordReader and RecordWriter Controller Service...
33. What are the NiFi processor scheduling strategies?
NiFi provides two scheduling strategies controlling when a processor's onTrigger method is invoked: Timer Driven (default): The processor is scheduled to run at a fixed time interval. The Run Schedule property sets the interval — 0 sec means run as fast as possible (yielding only for yield durati...
34. What is the difference between EvaluateJsonPath and FlattenJson processors?
EvaluateJsonPath and FlattenJson both work with JSON content but serve fundamentally different purposes. EvaluateJsonPath extracts specific values from a JSON payload using JSONPath expressions and writes those values either to FlowFile attributes or to the FlowFile content. It is a targeted extr...
35. How does NiFi integrate with Apache Hadoop and HDFS?
NiFi provides a suite of processors for reading from and writing to HDFS (Hadoop Distributed File System) and integrates with the broader Hadoop ecosystem including Hive and HBase. The integration uses standard Hadoop client libraries and respects Hadoop authentication (Simple or Kerberos). Key H...
36. What is the UpdateAttribute processor and how is its Advanced Mode used?
UpdateAttribute is one of the most versatile NiFi processors. In basic mode, each User-Defined Property becomes an attribute name, and its value (which can use NiFi Expression Language) becomes the new attribute value. Adding a property processed.timestamp with value ${now():format('yyyy-MM-dd HH...
37. How do you implement deduplication in a NiFi flow?
Deduplication — preventing the same data from being processed more than once — is a common requirement. NiFi provides several mechanisms depending on scale, performance requirements, and what constitutes a duplicate. DetectDuplicate processor : The simplest approach. It uses a Distributed Map Cac...
38. What is the HandleHttpRequest and HandleHttpResponse processor pair used for?
HandleHttpRequest and HandleHttpResponse implement an HTTP server inside NiFi, enabling NiFi to act as a web service endpoint that receives HTTP requests from external clients, processes them as FlowFiles through the flow, and returns HTTP responses. HandleHttpRequest : Starts an embedded Jetty H...
39. How does NiFi achieve guaranteed delivery and what are its durability guarantees?
NiFi's architecture is specifically designed to provide guaranteed delivery — once data enters NiFi, it will not be silently lost due to hardware failure, software crash, or network issues. Several design decisions work together to achieve this. Persistent connection queues : FlowFiles in connect...
40. What is the Funnel component in NiFi and when do you use it?
A Funnel is a NiFi canvas component that merges FlowFiles from multiple incoming connections into a single outgoing connection. It has no processing logic — it is purely a flow topology tool for consolidating multiple data paths into one without introducing a processor's overhead, scheduling, or ...
41. What is the difference between GetFile and ListFile + FetchFile processors?
Both approaches ingest files from a local filesystem, but they differ in architecture, parallelism, and operational characteristics. GetFile : The older, simpler, single-processor approach. It lists a directory, picks up files matching the configured filter, moves or deletes the source file atomi...
42. How does NiFi support schema evolution in data pipelines?
Schema evolution — handling changes in data structure without breaking pipelines — is supported primarily through the record model and schema registry integration. NiFi's record-aware processors use Schema Access Strategies on RecordReaders and RecordWriters: Infer Schema : The reader analyzes th...
43. What is the RouteText processor and how does it differ from RouteOnContent?
RouteText and RouteOnContent are both NiFi processors that route FlowFiles based on patterns found in content, but they operate at fundamentally different granularities. RouteOnContent : Evaluates the entire FlowFile content against configured regex patterns. If any pattern matches anywhere in th...
44. What performance tuning options are available in NiFi and what are common bottleneck patterns?
NiFi performance tuning operates at several levels: JVM heap, thread pool sizes, repository configuration, and per-processor settings. JVM Heap (bootstrap.conf) : The java.arg.2=-Xms and java.arg.3=-Xmx settings control heap. NiFi's content repository keeps content on disk, so heap is primarily c...
45. How does NiFi integrate with cloud storage services like Amazon S3?
NiFi provides a comprehensive set of processors for integrating with Amazon S3 available in the nifi-aws-nar extension. ListS3 : Lists objects in an S3 bucket (filtered by prefix and last modified date). Produces one FlowFile per S3 object with attributes: s3.bucket , s3.key , filename , s3.etag ...
