Integration / Apache Kafka Interview questions
Explain the execution flow of a Kafka Streams topology processing a record?
A Kafka Streams application defines a topology — a graph of processing nodes (source, transformations, sink) — and every record pulled from an input topic flows through that graph node by node before any result is produced.
flowchart TD
A[Record polled from source topic partition] --> B[Deserialized into a typed key/value pair]
B --> C[Passed to the first processor node in the topology]
C --> D{Node type}
D -- Stateless, e.g. filter/map --> E[Transform or drop the record, pass downstream]
D -- Stateful, e.g. aggregate/join --> F[Read/update local state store, emit result]
E --> G{More nodes downstream?}
F --> G
G -- Yes --> C
G -- No --> H[Final result serialized and written to output topic, if a sink is defined]
H --> I[If stateful, changelog topic updated to reflect the state change]
This per-record flow happens continuously as the application polls its input topics, and because the topology is defined once at startup (via the DSL's fluent builder or the lower-level Processor API), Kafka Streams can statically determine which internal topics (repartition, changelog) it needs to create before any data actually starts flowing, rather than discovering that requirement dynamically at runtime.
What must happen before a stateful operation like aggregate can emit a result for a record?
When does Kafka Streams determine which internal topics it needs to create?
More Related questions...
