AI / Apache Paimon Interview questions
Why should changelog-producer be enabled only when necessary?
Generating a full before/after changelog is not free: with none, Paimon just tracks new/changed values across snapshots, which is the cheapest option. Every other setting adds work on the write path specifically to reconstruct the "before" value for every changed key, which the merged table alone doesn't retain.
input only works safely when the input itself is already a full changelog (like raw CDC), because it just passes input records straight through without extra computation — cheap, but narrow in applicability. lookup and full-compaction both do real extra work: lookup queries current on-disk state before committing, adding write-side latency, while full-compaction ties changelog generation to a full compaction, trading it for coarser (batched) freshness instead.
If nothing downstream actually consumes a before/after changelog, leaving changelog-producer at none avoids that extra write-side cost entirely — it's a feature to turn on deliberately for a specific streaming consumer's needs, not a default to enable everywhere.
More Related questions...