Explain the execution flow of building a streaming lakehouse pipeline with Iceberg and Flink?

A streaming pipeline using Flink and Iceberg continuously ingests events, periodically committing them as new snapshots, while downstream analytical queries read the same table concurrently — combining low-latency ingestion with the same consistent, ACID-compliant table other batch and interactive tools can also safely query.

Flink's Iceberg sink integrates with Flink's own checkpointing mechanism: events are buffered and, at each checkpoint interval, written out as new Iceberg data files and committed as a new snapshot, tying Iceberg's transactional commit boundary to Flink's own exactly-once processing guarantees rather than committing on some arbitrary, independent schedule.

Because each checkpoint-triggered commit tends to produce relatively small files (proportional to how much data arrived within that interval), streaming Iceberg pipelines are a classic source of the small-files problem discussed earlier, which is why a mature streaming-to-Iceberg architecture typically pairs the streaming ingestion job with a separate, periodic batch compaction job — keeping ingestion latency low while still maintaining an efficient, well-organized file layout for the downstream analytical queries that read the same table.