AI / Apache Paimon Interview questions
1. What is Apache Paimon?
Apache Paimon is an open-source lake format for building a real-time Lakehouse architecture that supports both streaming and batch operations. It innovatively combines a data lake format with a Log-Structured Merge-tree (LSM), which lets it bring real-time streaming updates into...
2. What is the purpose of Paimon's Catalog abstraction?
A Catalog is how Paimon manages the table of contents and metadata for its tables, and it is the recommended way to access any Paimon table rather than pointing directly at table files. It gives compute engines a consistent way...
3. What are the four types of metastores Paimon catalogs support?
Paimon catalogs currently support four metastore backends, each storing metadata differently: Metastore Behavior filesystem (default) Stores both metadata and table files in the filesystem itself. hive Additionally stores metadata in the Hive metastore, so tables are directly visible from Hive....
4. Define a primary key table in Paimon?
A primary key table is a Paimon table created with one or more columns declared as its primary key, which lets you insert, update, or delete individual records instead of only appending new rows. Primary keys consist of columns whose...
5. What are the two main table types in Paimon?
Every Paimon table falls into one of two types, depending on whether a primary key is declared: Table type Characteristics Primary key table Supports insert, update, and delete; records are merged by primary key using a merge engine; high-performance streaming...
6. Describe a Paimon Snapshot?
A snapshot is a JSON file stored in a table's snapshot directory that captures the state of the table at one point in time. It records the schema file in use and a manifest list containing all the changes belonging...
7. What are Manifest files used for in Paimon?
Manifest files and manifest lists live in a table's manifest directory and record the changes to LSM data files and changelog files that belong to a given snapshot — specifically, which data files were created and which were deleted. A...
8. What is a Bucket in Paimon?
A bucket is the smallest storage unit Paimon reads and writes at — unpartitioned tables, or each partition within a partitioned table, are sub-divided into buckets to add extra structure for efficient querying. Each bucket directory holds its own LSM...
9. List the merge engines Paimon supports for primary key tables?
When Paimon receives two or more records sharing a primary key, the table's merge-engine property decides how they're combined into one: Merge engine Behavior Deduplicate (default) Keeps only the latest record; discards the rest. Partial Update Merges non-null fields from...
10. What is the default merge engine in Paimon?
The Deduplicate merge engine is Paimon's default. When the sink receives two or more records with the same primary key, Paimon keeps only the latest one and discards the others, so the primary key stays unique....
11. What is the purpose of the changelog-producer table property?
changelog-producer controls what shape of change stream a primary key table's writer generates for downstream streaming readers. By default ( none ), Paimon only exposes merged changes across snapshots — it can tell you a key's new value, but not...
12. How do you use the sequence.field option in Paimon?
sequence.field tells Paimon which column to trust for merge ordering instead of relying on input arrival order. By default, a primary key table merges records in the order they arrive — the last one in wins — but distributed writers...
13. What is Paimon's LSM tree used for?
Paimon adopts the LSM tree (Log-Structured Merge-tree) as the on-disk data structure inside every bucket, which is what makes high-throughput streaming updates into a data lake practical in the first place. New records are first buffered in memory; when the...
14. What are Sorted Runs in a Paimon LSM tree?
A sorted run is one or more data files where records are sorted by primary key, and, crucially, primary key ranges never overlap within the same sorted run. Each data file belongs to exactly one sorted run....
15. Describe what Paimon's system tables are for?
Paimon ships a rich set of system tables that expose metadata about a table's own history and status through ordinary SQL, so you don't need a separate tool to inspect internals. Flink, Spark, Trino, and StarRocks can all query them....
16. How do you use the $snapshots system table?
Querying my_table$snapshots returns the history of every snapshot the table has produced, including the commit user, commit time, commit kind, and record counts for that snapshot. SELECT * FROM my_table$snapshots; It's the table to check first when you need to...
17. What is the purpose of Tags in Paimon?
Tags solve a specific problem: snapshots are convenient for querying historical data, but tables expire old snapshots on a schedule, and expiration deletes the underlying data files too. A tag pins a chosen snapshot's manifests and data files so they...
18. What is an Append Table in Paimon?
An append table is any Paimon table created without a primary key. It cannot receive changelogs or be upserted directly — it can only accept incoming append data, much like a plain Hive partitioned table....
19. How do you apply schema evolution during CDC ingestion into Paimon?
Paimon's CDC ingestion tooling can keep a target table's schema in sync with a changing source automatically, but only through the right ingestion path. Plain Flink SQL CDC ingestion does not propagate source schema changes to Paimon....
20. What are the consistency guarantees Paimon provides for writers?
Paimon writers use a two-phase commit protocol to atomically commit a batch of records to a table, so a snapshot either fully appears or doesn't appear at all — readers never see a half-written commit. For concurrency between writers: two...
21. What is the purpose of the Audit Log system table?
The $audit_log system table exposes a table's incremental changelog with an extra rowkind column attached, so you can filter and audit exactly what kind of change each row represents. rowkind takes one of four values: +I for insertion, -U for...
22. Describe the Read-optimized (ro) system table?
The $ro system table trades a small amount of freshness for a large amount of read speed: it improves query performance by only scanning files that don't need merging at query time. On a primary key table, that means $ro...
23. Why does Paimon combine a lake format with an LSM-tree structure?
Traditional data lake formats are built around immutable, append-only files, which makes them cheap to scan but expensive to update — a single-row change historically meant rewriting whole files. Traditional databases handle updates well but weren't designed to scale to...
24. How does the Deduplicate merge engine handle a DELETE record?
Under the default Deduplicate merge engine, if the newest record Paimon sees for a given primary key is a DELETE , that key's earlier records are removed entirely from the merged result — the key effectively disappears from the table....
25. What is the difference between the Partial Update and Aggregation merge engines?
Both combine multiple records that share a primary key field-by-field, but they combine values very differently: Merge engine Rule per field Typical use case Partial Update Non-null incoming values overwrite the existing value; nulls are ignored, preserving the old value....
26. What is the difference between Fixed Bucket and Dynamic Bucket modes?
Both decide which bucket a record lands in, but they solve different scaling problems: Mode How buckets are assigned Best for Fixed Bucket A fixed bucket count set with bucket = N ; assignment is a deterministic hash of the...
27. When should you choose Postpone Bucket mode?
Postpone Bucket mode is for a specific pattern: you want records to land quickly with low latency, but you don't yet want to pay the cost of actually deciding buckets and merging primary key data. Records are written to a...
28. What happens when Paimon's sorted runs contain overlapping primary key ranges?
Overlap between sorted runs is expected and normal — it's how new writes get in without rewriting old files. The cost shows up at read time: a query has to open every sorted run whose key range could contain the...
29. Why should changelog-producer be enabled only when necessary?
Generating a full before/after changelog is not free: with none , Paimon just tracks new/changed values across snapshots, which is the cheapest option. Every other setting adds work on the write path specifically to reconstruct the "before" value for every...
30. What is the difference between the input and lookup changelog producers?
Both aim to give downstream consumers a full before/after changelog, but they get the "before" value from different places: Producer Where the before value comes from Requirement input Passed straight through from the input stream, unmodified. The input must already...
31. How does the full-compaction changelog producer differ from lookup?
Both eventually give you a complete before/after changelog, but they differ in when that changelog is produced and how fresh it is. lookup reconstructs the before-value at write time, on every commit, so changelog records appear about as fast as...
32. Why do concurrent writers to the same partition only get snapshot isolation instead of full isolation?
Paimon's two-phase commit protects atomicity — a snapshot appears completely or not at all — but it doesn't serialize every writer against every other writer touching the same partition. Doing so would mean forcing concurrent writers into a single queue,...
33. When should you choose the Hive catalog over the filesystem catalog?
The filesystem catalog stores both metadata and data purely in the filesystem, which is the simplest choice when Paimon is the only tool that needs to know about your tables. It doesn't require any external metastore service to run....
34. How can you optimize primary key lookups using the bucket-key option?
By default, a primary key table buckets records by hashing the entire primary key. If your primary key has multiple columns but most lookups actually filter on just a subset of them, hashing the full key spreads matching rows across...
35. What is the difference between the sequence.field and rowkind.field options?
Both influence how Paimon merges records for the same primary key, but they control different aspects: Option Controls sequence.field Which column decides merge order when records arrive out of order (e.g. an event-time or version column)....
36. Explain the execution flow of a two-phase commit when a Paimon writer flushes data?
Paimon writers rely on a classic two-phase commit to guarantee that a snapshot is all-or-nothing, even across a distributed engine like Flink: sequenceDiagram participant Writer as Writer task(s) participant Coord as Commit coordinator participant Store as Table (snapshot dir) Writer->>Writer:...
37. Why is Cross Partitions Upsert more expensive than a normal bucketed upsert?
In a standard Paimon primary key table, a record's partition and bucket are derived deterministically from its columns, so an upsert only ever needs to check for an existing matching key inside that one target bucket — a cheap, local...
38. How do you troubleshoot excessive small files from streaming writes into a Paimon table?
Streaming writes commit frequently and in small batches, which naturally produces many small data files — left unmanaged, this degrades both read performance (more files to open and merge) and puts pressure on the underlying filesystem/object store. Check the $files...
39. What is the difference between the audit_log and binlog system tables?
Both expose row-level change information, but they differ in shape and intended consumer: System table Shape Best for $audit_log One row per change event, with a single added rowkind column (+I/-U/+U/-D). General auditing and debugging where you want to filter...
40. Explain the internal working of automatic tag creation with a watermark?
Watermark-based automatic tagging lets Paimon create tags aligned to event time rather than wall-clock commit time, which matters for streaming pipelines where records can arrive late. flowchart TD A["Streaming writer commits new snapshot"] --> B["Snapshot records its watermark (max event-time...
41. When should you choose the REST catalog over the Hive catalog?
The REST catalog gives you a single, engine-agnostic HTTP interface to a catalog's metadata, rather than requiring every client to speak the Hive Thrift protocol directly. It's the better choice when you want centralized catalog access across a heterogeneous mix...
42. What happens when you roll back a Paimon table to an earlier tag?
Rolling back moves a table's current state back to exactly what a given tag (or snapshot) captured, discarding everything that happened afterward as the table's "current" view. Because the tag already protected that point-in-time's manifests and data files from expiration,...
43. How does Paimon achieve streaming-batch unification on the same table?
Paimon treats a table's snapshot history as both a bounded dataset and an unbounded stream, depending on how you read it, without needing two separate copies of the data. A batch job simply reads the latest (or a tagged/time-traveled) snapshot...
44. Why does Paimon recommend keeping bucket data size between 200MB and 1GB?
Bucket count directly caps write parallelism and shapes file sizes, so it sits on a tradeoff between two failure modes. Too many buckets for the actual data volume means each bucket ends up holding very little data, which produces lots...
45. What is the difference between the First Row and Deduplicate merge engines?
Both are simple "pick one record" merge engines, but they pick from opposite ends of the arrival order: Merge engine Keeps Typical use Deduplicate The latest record seen for a key. Standard upsert semantics, mirroring a source database's current state....
46. How can you optimize query performance using the read-optimized system table?
When a downstream query can tolerate slightly stale data — as of the last full compaction rather than the absolute latest commit — querying my_table$ro instead of the base table skips the merge-at-read-time cost entirely, since it only touches files...
47. What is the difference between Paimon's and Apache Iceberg's core design philosophy?
Both are open table formats supporting ACID transactions, schema evolution, and time travel on top of object storage, but they optimize for different primary workloads: Aspect Apache Paimon Apache Iceberg Core storage engine LSM-tree per bucket, designed for frequent streaming...
48. Why did Apache Paimon originally start as part of the Flink project, and what is it called now?
Paimon began life inside the Flink community as FLIP-188, "Introduce Built-in Dynamic Table Storage," shipping as Flink Table Store . The motivation was specifically to give Flink's streaming SQL a native storage layer that could hold large, continuously updated tables...
49. Which is better and why: lookup or full-compaction changelog producer for a 30-minute-latency pipeline?
For a pipeline whose consumers only need updates roughly every 30 minutes, full-compaction is generally the better fit, precisely because its coarser cadence matches (rather than wastes) the freshness the pipeline actually needs. lookup reconstructs the before-value on every single...
50. How do you troubleshoot duplicate rows appearing when using Dynamic Bucket mode with multiple concurrent write jobs?
Dynamic Bucket mode relies on an internal index mapping each primary key to the bucket it belongs in, so that all records for a given key are routed consistently to the same bucket regardless of when they arrive. That guarantee...
