AI / Apache Paimon Interview questions
How does the full-compaction changelog producer differ from lookup?
Both eventually give you a complete before/after changelog, but they differ in when that changelog is produced and how fresh it is. lookup reconstructs the before-value at write time, on every commit, so changelog records appear about as fast as the underlying commits do.
full-compaction instead ties changelog generation to a full compaction of the LSM tree: Paimon compares the table's state before and after a full compaction and emits the difference as the changelog. Because full compactions don't run on every single commit — they're triggered periodically, via full-compaction.delta-commits or a time-based trigger — changelog freshness under this mode is coarser and batched rather than continuous.
CREATE TABLE my_table (...) WITH ( 'changelog-producer' = 'full-compaction', 'full-compaction.delta-commits' = '5' );
Pick lookup when downstream consumers need low-latency changelog delivery; pick full-compaction when a periodic, less frequent changelog is acceptable and you'd rather batch the extra compute cost into scheduled compactions instead of paying it on every write.
More Related questions...