AI / Apache Paimon Interview questions
Explain the execution flow of a two-phase commit when a Paimon writer flushes data?
Paimon writers rely on a classic two-phase commit to guarantee that a snapshot is all-or-nothing, even across a distributed engine like Flink:
sequenceDiagram
participant Writer as Writer task(s)
participant Coord as Commit coordinator
participant Store as Table (snapshot dir)
Writer->>Writer: Buffer records, sort, flush as new data files (Phase 1: PRECOMMIT)
Writer->>Coord: Report written file paths + stats
Coord->>Coord: Collect commits from ALL writer tasks
Coord->>Store: Atomically write new snapshot file referencing all files (Phase 2: COMMIT)
Store-->>Coord: Snapshot visible to readers
Coord-->>Writer: Commit acknowledged
In phase one, each writer task independently flushes its own buffered records to new data files on disk and reports what it wrote, without yet making those files visible as part of the table. In phase two, once every relevant writer task has reported in, a single coordinator atomically writes a new snapshot file that references all of the newly written files at once.
Because the snapshot file write is atomic, readers either see the old snapshot (none of the new files) or the new snapshot (all of the new files) — there is no window where only some writer tasks' files are visible. If any writer task fails before reporting in, the whole commit is abandoned and no partial snapshot is ever published.
More Related questions...