What is Apache Iceberg?

Apache Iceberg is an open-source table format for large, analytical datasets stored in a data lake, adding database-like structure — schema, partitioning, snapshots, ACID transactions — on top of files (typically Parquet) sitting in object storage like S3, GCS, or HDFS.

It originated at Netflix, where engineers needed a way to reliably manage petabyte-scale tables in a data lake without the correctness and performance problems that plagued Hive-style table management, and it was later donated to the Apache Software Foundation, where it's now a top-level project maintained by an active open-source community.

Rather than being a storage engine or query engine itself, Iceberg is a specification and a set of libraries that any compatible engine — Spark, Flink, Trino, Presto, Hive, Dremio, and others — can read and write against, which is what lets multiple different tools operate on the exact same underlying table consistently rather than each tool needing its own separate, incompatible way of tracking table state.