AI / Apache Paimon Interview questions
1. What is Apache Paimon?
Apache Paimon is an open-source lake format for building a real-time Lakehouse architecture that supports both streaming and batch operations. It innovatively combines a data lake format with a Log-Structured Merge-tree (LSM), which lets it bring real-time streaming updates into lake storage inst...
2. What is the purpose of Paimon's Catalog abstraction?
A Catalog is how Paimon manages the table of contents and metadata for its tables, and it is the recommended way to access any Paimon table rather than pointing directly at table files. It gives compute engines a consistent way to discover databases and tables, resolve schemas, and locate the und...
3. What are the four types of metastores Paimon catalogs support?
Paimon catalogs currently support four metastore backends, each storing metadata differently: Metastore Behavior filesystem (default) Stores both metadata and table files in the filesystem itself. hive Additionally stores metadata in the Hive metastore, so tables are directly visible from Hive. j...
4. Define a primary key table in Paimon?
A primary key table is a Paimon table created with one or more columns declared as its primary key, which lets you insert, update, or delete individual records instead of only appending new rows. Primary keys consist of columns whose combined values are unique per record. Paimon enforces ordering...
5. What are the two main table types in Paimon?
Every Paimon table falls into one of two types, depending on whether a primary key is declared: Table type Characteristics Primary key table Supports insert, update, and delete; records are merged by primary key using a merge engine; high-performance streaming upserts. Append table (no primary ke...
6. Describe a Paimon Snapshot?
A snapshot is a JSON file stored in a table's snapshot directory that captures the state of the table at one point in time. It records the schema file in use and a manifest list containing all the changes belonging to that snapshot. Reading the latest snapshot gives you the current state of the t...
7. What are Manifest files used for in Paimon?
Manifest files and manifest lists live in a table's manifest directory and record the changes to LSM data files and changelog files that belong to a given snapshot — specifically, which data files were created and which were deleted. A manifest list is simply a list of manifest file names; ...
8. What is a Bucket in Paimon?
A bucket is the smallest storage unit Paimon reads and writes at — unpartitioned tables, or each partition within a partitioned table, are sub-divided into buckets to add extra structure for efficient querying. Each bucket directory holds its own LSM tree and changelog files. Which bucket a...
9. List the merge engines Paimon supports for primary key tables?
When Paimon receives two or more records sharing a primary key, the table's merge-engine property decides how they're combined into one: Merge engine Behavior Deduplicate (default) Keeps only the latest record; discards the rest. Partial Update Merges non-null fields from multiple records column ...
10. What is the default merge engine in Paimon?
The Deduplicate merge engine is Paimon's default. When the sink receives two or more records with the same primary key, Paimon keeps only the latest one and discards the others, so the primary key stays unique. If the latest record for a key happens to be a DELETE , every prior record with that k...
11. What is the purpose of the changelog-producer table property?
changelog-producer controls what shape of change stream a primary key table's writer generates for downstream streaming readers. By default ( none ), Paimon only exposes merged changes across snapshots — it can tell you a key's new value, but not its old one. Some streaming consumers, such ...
12. How do you use the sequence.field option in Paimon?
sequence.field tells Paimon which column to trust for merge ordering instead of relying on input arrival order. By default, a primary key table merges records in the order they arrive — the last one in wins — but distributed writers can deliver records out of order. CREATE TABLE my_ta...
13. What is Paimon's LSM tree used for?
Paimon adopts the LSM tree (Log-Structured Merge-tree) as the on-disk data structure inside every bucket, which is what makes high-throughput streaming updates into a data lake practical in the first place. New records are first buffered in memory; when the buffer fills, they're sorted and flushe...
14. What are Sorted Runs in a Paimon LSM tree?
A sorted run is one or more data files where records are sorted by primary key, and, crucially, primary key ranges never overlap within the same sorted run. Each data file belongs to exactly one sorted run. Different sorted runs, however, can and do have overlapping key ranges — the same ke...
15. Describe what Paimon's system tables are for?
Paimon ships a rich set of system tables that expose metadata about a table's own history and status through ordinary SQL, so you don't need a separate tool to inspect internals. Flink, Spark, Trino, and StarRocks can all query them. They come in two families: data system tables (queried per tabl...
16. How do you use the $snapshots system table?
Querying my_table$snapshots returns the history of every snapshot the table has produced, including the commit user, commit time, commit kind, and record counts for that snapshot. SELECT * FROM my_table$snapshots; It's the table to check first when you need to know what happened and when: which s...
17. What is the purpose of Tags in Paimon?
Tags solve a specific problem: snapshots are convenient for querying historical data, but tables expire old snapshots on a schedule, and expiration deletes the underlying data files too. A tag pins a chosen snapshot's manifests and data files so they survive that expiration. A common pattern is c...
18. What is an Append Table in Paimon?
An append table is any Paimon table created without a primary key. It cannot receive changelogs or be upserted directly — it can only accept incoming append data, much like a plain Hive partitioned table. CREATE TABLE my_table ( product_id BIGINT, price DOUBLE, sales BIGINT ) WITH ( -- 'tar...
19. How do you apply schema evolution during CDC ingestion into Paimon?
Paimon's CDC ingestion tooling can keep a target table's schema in sync with a changing source automatically, but only through the right ingestion path. Plain Flink SQL CDC ingestion does not propagate source schema changes to Paimon. Using the dedicated sync actions instead — such as MySql...
20. What are the consistency guarantees Paimon provides for writers?
Paimon writers use a two-phase commit protocol to atomically commit a batch of records to a table, so a snapshot either fully appears or doesn't appear at all — readers never see a half-written commit. For concurrency between writers: two writers modifying different partitions at the same t...
21. What is the purpose of the Audit Log system table?
The $audit_log system table exposes a table's incremental changelog with an extra rowkind column attached, so you can filter and audit exactly what kind of change each row represents. rowkind takes one of four values: +I for insertion, -U for the previous content of an updated row, +U for the new...
22. Describe the Read-optimized (ro) system table?
The $ro system table trades a small amount of freshness for a large amount of read speed: it improves query performance by only scanning files that don't need merging at query time. On a primary key table, that means $ro only scans files at the topmost LSM level, so it reflects the result of the ...
23. Why does Paimon combine a lake format with an LSM-tree structure?
Traditional data lake formats are built around immutable, append-only files, which makes them cheap to scan but expensive to update — a single-row change historically meant rewriting whole files. Traditional databases handle updates well but weren't designed to scale to lake-sized batch and...
24. How does the Deduplicate merge engine handle a DELETE record?
Under the default Deduplicate merge engine, if the newest record Paimon sees for a given primary key is a DELETE , that key's earlier records are removed entirely from the merged result — the key effectively disappears from the table. CREATE TABLE my_table ( pk BIGINT PRIMARY KEY NOT ENFORC...
25. What is the difference between the Partial Update and Aggregation merge engines?
Both combine multiple records that share a primary key field-by-field, but they combine values very differently: Merge engine Rule per field Typical use case Partial Update Non-null incoming values overwrite the existing value; nulls are ignored, preserving the old value. Assembling one wide row ...
26. What is the difference between Fixed Bucket and Dynamic Bucket modes?
Both decide which bucket a record lands in, but they solve different scaling problems: Mode How buckets are assigned Best for Fixed Bucket A fixed bucket count set with bucket = N ; assignment is a deterministic hash of the bucket key. Predictable data volume where you can size buckets up front; ...
27. When should you choose Postpone Bucket mode?
Postpone Bucket mode is for a specific pattern: you want records to land quickly with low latency, but you don't yet want to pay the cost of actually deciding buckets and merging primary key data. Records are written to a special bucket = -2 placeholder area immediately, and the real bucketing an...
28. What happens when Paimon's sorted runs contain overlapping primary key ranges?
Overlap between sorted runs is expected and normal — it's how new writes get in without rewriting old files. The cost shows up at read time: a query has to open every sorted run whose key range could contain the requested key(s), then merge whatever records it finds for each key using the t...
29. Why should changelog-producer be enabled only when necessary?
Generating a full before/after changelog is not free: with none , Paimon just tracks new/changed values across snapshots, which is the cheapest option. Every other setting adds work on the write path specifically to reconstruct the "before" value for every changed key, which the merged table alon...
30. What is the difference between the input and lookup changelog producers?
Both aim to give downstream consumers a full before/after changelog, but they get the "before" value from different places: Producer Where the before value comes from Requirement input Passed straight through from the input stream, unmodified. The input must already be a complete changelog (e.g. ...
31. How does the full-compaction changelog producer differ from lookup?
Both eventually give you a complete before/after changelog, but they differ in when that changelog is produced and how fresh it is. lookup reconstructs the before-value at write time, on every commit, so changelog records appear about as fast as the underlying commits do. full-compaction instead ...
32. Why do concurrent writers to the same partition only get snapshot isolation instead of full isolation?
Paimon's two-phase commit protects atomicity — a snapshot appears completely or not at all — but it doesn't serialize every writer against every other writer touching the same partition. Doing so would mean forcing concurrent writers into a single queue, which would badly limit throug...
33. When should you choose the Hive catalog over the filesystem catalog?
The filesystem catalog stores both metadata and data purely in the filesystem, which is the simplest choice when Paimon is the only tool that needs to know about your tables. It doesn't require any external metastore service to run. The Hive catalog additionally registers tables in the Hive metas...
34. How can you optimize primary key lookups using the bucket-key option?
By default, a primary key table buckets records by hashing the entire primary key. If your primary key has multiple columns but most lookups actually filter on just a subset of them, hashing the full key spreads matching rows across every bucket — forcing a scan of buckets that don't actual...
35. What is the difference between the sequence.field and rowkind.field options?
Both influence how Paimon merges records for the same primary key, but they control different aspects: Option Controls sequence.field Which column decides merge order when records arrive out of order (e.g. an event-time or version column). rowkind.field Which column tells Paimon the record's chan...
36. Explain the execution flow of a two-phase commit when a Paimon writer flushes data?
Paimon writers rely on a classic two-phase commit to guarantee that a snapshot is all-or-nothing, even across a distributed engine like Flink: sequenceDiagram participant Writer as Writer task(s) participant Coord as Commit coordinator participant Store as Table (snapshot dir) Writer->>Writer: Bu...
37. Why is Cross Partitions Upsert more expensive than a normal bucketed upsert?
In a standard Paimon primary key table, a record's partition and bucket are derived deterministically from its columns, so an upsert only ever needs to check for an existing matching key inside that one target bucket — a cheap, local operation. Cross Partitions Upsert mode exists for a hard...
38. How do you troubleshoot excessive small files from streaming writes into a Paimon table?
Streaming writes commit frequently and in small batches, which naturally produces many small data files — left unmanaged, this degrades both read performance (more files to open and merge) and puts pressure on the underlying filesystem/object store. Check the $files system table. Query my_t...
39. What is the difference between the audit_log and binlog system tables?
Both expose row-level change information, but they differ in shape and intended consumer: System table Shape Best for $audit_log One row per change event, with a single added rowkind column (+I/-U/+U/-D). General auditing and debugging where you want to filter or count changes by type in plain SQ...
40. Explain the internal working of automatic tag creation with a watermark?
Watermark-based automatic tagging lets Paimon create tags aligned to event time rather than wall-clock commit time, which matters for streaming pipelines where records can arrive late. flowchart TD A["Streaming writer commits new snapshot"] --> B["Snapshot records its watermark (max event-time se...
41. When should you choose the REST catalog over the Hive catalog?
The REST catalog gives you a single, engine-agnostic HTTP interface to a catalog's metadata, rather than requiring every client to speak the Hive Thrift protocol directly. It's the better choice when you want centralized catalog access across a heterogeneous mix of engines and languages without d...
42. What happens when you roll back a Paimon table to an earlier tag?
Rolling back moves a table's current state back to exactly what a given tag (or snapshot) captured, discarding everything that happened afterward as the table's "current" view. Because the tag already protected that point-in-time's manifests and data files from expiration, the rollback can comple...
43. How does Paimon achieve streaming-batch unification on the same table?
Paimon treats a table's snapshot history as both a bounded dataset and an unbounded stream, depending on how you read it, without needing two separate copies of the data. A batch job simply reads the latest (or a tagged/time-traveled) snapshot as a fixed, bounded result; a streaming job instead s...
44. Why does Paimon recommend keeping bucket data size between 200MB and 1GB?
Bucket count directly caps write parallelism and shapes file sizes, so it sits on a tradeoff between two failure modes. Too many buckets for the actual data volume means each bucket ends up holding very little data, which produces lots of small files — more files for readers to open and mer...
45. What is the difference between the First Row and Deduplicate merge engines?
Both are simple "pick one record" merge engines, but they pick from opposite ends of the arrival order: Merge engine Keeps Typical use Deduplicate The latest record seen for a key. Standard upsert semantics, mirroring a source database's current state. First Row The first record ever seen for a k...
46. How can you optimize query performance using the read-optimized system table?
When a downstream query can tolerate slightly stale data — as of the last full compaction rather than the absolute latest commit — querying my_table$ro instead of the base table skips the merge-at-read-time cost entirely, since it only touches files that are already fully merged. -- S...
47. What is the difference between Paimon's and Apache Iceberg's core design philosophy?
Both are open table formats supporting ACID transactions, schema evolution, and time travel on top of object storage, but they optimize for different primary workloads: Aspect Apache Paimon Apache Iceberg Core storage engine LSM-tree per bucket, designed for frequent streaming upserts Immutable P...
48. Why did Apache Paimon originally start as part of the Flink project, and what is it called now?
Paimon began life inside the Flink community as FLIP-188, "Introduce Built-in Dynamic Table Storage," shipping as Flink Table Store . The motivation was specifically to give Flink's streaming SQL a native storage layer that could hold large, continuously updated tables efficiently — somethi...
49. Which is better and why: lookup or full-compaction changelog producer for a 30-minute-latency pipeline?
For a pipeline whose consumers only need updates roughly every 30 minutes, full-compaction is generally the better fit, precisely because its coarser cadence matches (rather than wastes) the freshness the pipeline actually needs. lookup reconstructs the before-value on every single commit, which ...
50. How do you troubleshoot duplicate rows appearing when using Dynamic Bucket mode with multiple concurrent write jobs?
Dynamic Bucket mode relies on an internal index mapping each primary key to the bucket it belongs in, so that all records for a given key are routed consistently to the same bucket regardless of when they arrive. That guarantee assumes a single logical writer path is responsible for assigning and...
